{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Rent Listing Inqueries Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Rental Listing Inquiries数据集是Kaggle平台上的一个分类竞赛任务，需要根据公寓的特征来预测其受欢迎程度（用户感兴趣程度分为高、中、低三类）。其中房屋的特征x共有14维，响应值y为用户对该公寓的感兴趣程度。评价标准为logloss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.参数调优（细调）：max_depth & min_child_weight  \n",
    "(粗调，参数的步长为2；下一步是在粗调最佳参数周围，将步长降为1，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV进行两个参数的同时调优  \n",
    "注：GridSearchCV中最终判定的是scoring越大的模型越好，若用log_loss作为评价指标，在函数中要用neg_log_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "      <th>interest_level</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.5</td>\n",
       "      <td>3</td>\n",
       "      <td>3000</td>\n",
       "      <td>1200.0</td>\n",
       "      <td>750.000000</td>\n",
       "      <td>-1.5</td>\n",
       "      <td>4.5</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>5465</td>\n",
       "      <td>2732.5</td>\n",
       "      <td>1821.666667</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.0</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3275</td>\n",
       "      <td>1637.5</td>\n",
       "      <td>1637.500000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>18</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.0</td>\n",
       "      <td>4</td>\n",
       "      <td>3350</td>\n",
       "      <td>1675.0</td>\n",
       "      <td>670.000000</td>\n",
       "      <td>-3.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>28</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 228 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.5         3   3000           1200.0      750.000000       -1.5   \n",
       "1        1.0         2   5465           2732.5     1821.666667       -1.0   \n",
       "2        1.0         1   2850           1425.0     1425.000000        0.0   \n",
       "3        1.0         1   3275           1637.5     1637.500000        0.0   \n",
       "4        1.0         4   3350           1675.0      670.000000       -3.0   \n",
       "\n",
       "   room_num  Year  Month  Day       ...        walk  walls  war  washer  \\\n",
       "0       4.5  2016      6   24       ...           0      0    0       0   \n",
       "1       3.0  2016      6   12       ...           0      0    0       0   \n",
       "2       2.0  2016      4   17       ...           0      0    0       0   \n",
       "3       2.0  2016      4   18       ...           0      0    0       0   \n",
       "4       5.0  2016      4   28       ...           0      0    1       0   \n",
       "\n",
       "   water  wheelchair  wifi  windows  work  interest_level  \n",
       "0      0           0     0        0     0               1  \n",
       "1      0           0     0        0     0               2  \n",
       "2      0           0     0        0     0               0  \n",
       "3      0           0     0        0     0               2  \n",
       "4      0           0     0        0     0               2  \n",
       "\n",
       "[5 rows x 228 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"RentListingInquries_FE_train.csv\")\n",
    "test = pd.read_csv(dpath +\"RentListingInquries_FE_test.csv\")\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>bathrooms</th>\n",
       "      <th>bedrooms</th>\n",
       "      <th>price</th>\n",
       "      <th>price_bathrooms</th>\n",
       "      <th>price_bedrooms</th>\n",
       "      <th>room_diff</th>\n",
       "      <th>room_num</th>\n",
       "      <th>Year</th>\n",
       "      <th>Month</th>\n",
       "      <th>Day</th>\n",
       "      <th>...</th>\n",
       "      <th>virtual</th>\n",
       "      <th>walk</th>\n",
       "      <th>walls</th>\n",
       "      <th>war</th>\n",
       "      <th>washer</th>\n",
       "      <th>water</th>\n",
       "      <th>wheelchair</th>\n",
       "      <th>wifi</th>\n",
       "      <th>windows</th>\n",
       "      <th>work</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>2950</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>1475.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2850</td>\n",
       "      <td>1425.000000</td>\n",
       "      <td>950.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "      <td>3758</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>1879.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>2</td>\n",
       "      <td>3300</td>\n",
       "      <td>1650.000000</td>\n",
       "      <td>1100.000000</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>6</td>\n",
       "      <td>11</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>2</td>\n",
       "      <td>4900</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>1633.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2016</td>\n",
       "      <td>4</td>\n",
       "      <td>12</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 227 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   bathrooms  bedrooms  price  price_bathrooms  price_bedrooms  room_diff  \\\n",
       "0        1.0         1   2950      1475.000000     1475.000000        0.0   \n",
       "1        1.0         2   2850      1425.000000      950.000000       -1.0   \n",
       "2        1.0         1   3758      1879.000000     1879.000000        0.0   \n",
       "3        1.0         2   3300      1650.000000     1100.000000       -1.0   \n",
       "4        2.0         2   4900      1633.333333     1633.333333        0.0   \n",
       "\n",
       "   room_num  Year  Month  Day  ...   virtual  walk  walls  war  washer  water  \\\n",
       "0       2.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "1       3.0  2016      6   24  ...         0     0      0    1       0      0   \n",
       "2       2.0  2016      6    3  ...         0     0      0    0       0      0   \n",
       "3       3.0  2016      6   11  ...         0     0      0    0       0      0   \n",
       "4       4.0  2016      4   12  ...         0     0      0    1       0      0   \n",
       "\n",
       "   wheelchair  wifi  windows  work  \n",
       "0           0     0        0     0  \n",
       "1           0     0        0     0  \n",
       "2           0     0        0     0  \n",
       "3           1     0        0     0  \n",
       "4           0     0        0     0  \n",
       "\n",
       "[5 rows x 227 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "发现测试集中没有y，这次作业无法得到在测试集上的表现情况"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "*********** train **********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 49352 entries, 0 to 49351\n",
      "Columns: 228 entries, bathrooms to interest_level\n",
      "dtypes: float64(9), int64(219)\n",
      "memory usage: 85.8 MB\n",
      "************ test ***********\n",
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 74659 entries, 0 to 74658\n",
      "Columns: 227 entries, bathrooms to work\n",
      "dtypes: float64(9), int64(218)\n",
      "memory usage: 129.3 MB\n"
     ]
    }
   ],
   "source": [
    "print(\"*********** train **********\")\n",
    "train.info()\n",
    "print(\"************ test ***********\")\n",
    "test.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "测试集中少一维，即y标签"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAELCAYAAAARNxsIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAGx5JREFUeJzt3X2wHXWd5/H3h/AgPibIhcokYYKaGUUdo14hypSj6EDAHRMpUChHIkNtxAKF0rUEd5aMPMyM4ygrM8pupogkrkOIiBI1GDMMyII8JGB4CJHlioxcyZJgeNQVivDZP/p35eTm5N7OpU9OTu7nVdV1ur/96z7fwy340v379a9lm4iIiCbs0e0EIiJi95GiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIas2e3E9jZ9t9/f0+fPr3baURE9JTbbrvtEdt9o7Ubd0Vl+vTprFmzpttpRET0FEn/Uaddbn9FRERjOlZUJL1I0q2S7pC0TtLnS/xSSb+QtLYsM0tcki6SNCDpTklvaTnXPEn3lWVeS/ytku4qx1wkSZ36PRERMbpO3v56GjjC9lOS9gJukHR12fcZ21cMa380MKMshwEXA4dJ2g9YAPQDBm6TtNz2o6XNfOBmYAUwG7iaiIjoio5dqbjyVNncqywjzbM/B1hSjrsZmChpMnAUsMr25lJIVgGzy76X277J1fz9S4C5nfo9ERExuo72qUiaIGktsJGqMNxSdl1QbnFdKGmfEpsCPNhy+GCJjRQfbBNvl8d8SWskrdm0adML/l0REdFeR4uK7S22ZwJTgUMlvQE4G3gt8DZgP+CzpXm7/hCPId4uj4W2+2339/WNOiIuIiLGaKeM/rL9GHAdMNv2hnKL62ng68ChpdkgMK3lsKnAQ6PEp7aJR0REl3Ry9FefpIllfV/gvcDPSl8IZaTWXODucshy4KQyCmwW8LjtDcBK4EhJkyRNAo4EVpZ9T0qaVc51EnBVp35PRESMrpOjvyYDiyVNoCpey2x/X9K/S+qjun21Fji1tF8BHAMMAL8FTgawvVnSecDq0u5c25vL+seBS4F9qUZ9ZeRXREQXqRo4NX709/c7T9RH7NoO/6fDu53Cbu/GT9y4Q+0l3Wa7f7R2eaI+IiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYmIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMR0rKpJeJOlWSXdIWifp8yV+sKRbJN0n6XJJe5f4PmV7oOyf3nKus0v8XklHtcRnl9iApLM69VsiIqKeTl6pPA0cYftNwExgtqRZwBeAC23PAB4FTintTwEetf0a4MLSDkmHACcArwdmA1+TNEHSBOCrwNHAIcCJpW1ERHRJx4qKK0+Vzb3KYuAI4IoSXwzMLetzyjZl/3skqcSX2n7a9i+AAeDQsgzYvt/2M8DS0jYiIrqko30q5YpiLbARWAX8HHjM9rOlySAwpaxPAR4EKPsfB17ZGh92zPbiERHRJR0tKra32J4JTKW6snhdu2blU9vZt6PxbUiaL2mNpDWbNm0aPfGIiBiTnTL6y/ZjwHXALGCipD3LrqnAQ2V9EJgGUPa/AtjcGh92zPbi7b5/oe1+2/19fX1N/KSIiGijk6O/+iRNLOv7Au8F1gPXAseVZvOAq8r68rJN2f/vtl3iJ5TRYQcDM4BbgdXAjDKabG+qzvzlnfo9ERExuj1HbzJmk4HFZZTWHsAy29+XdA+wVNL5wE+BS0r7S4BvSBqgukI5AcD2OknLgHuAZ4HTbG8BkHQ6sBKYACyyva6DvyciIkbRsaJi+07gzW3i91P1rwyP/w44fjvnugC4oE18BbDiBScbERGNyBP1ERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjUlQiIqIxKSoREdGYFJWIiGjMqEVF0ksk7VHW/0jS+yXt1fnUIiKi19S5UrkeeJGkKcA1wMnApZ1MKiIielOdoiLbvwWOBf7J9geo3gkfERGxlVpFRdLbgQ8DPyixTk6ZHxERPapOUTkTOBv4Tnm3yauoXrQVERGxlVGvOGz/GPixpJeU7fuBT3Y6sYiI6D11Rn+9vbytcX3ZfpOkr3U8s4iI6Dl1bn/9d+Ao4NcAtu8A3tnJpCIiojfVevjR9oPDQls6kEtERPS4OqO4HpT0DsCS9qbqT1nf2bQiIqIX1blSORU4DZgCDAIzy3ZERMRWRi0qth+x/WHbB9o+wPZf2v71aMdJmibpWknrJa2TdEaJ/42kX0laW5ZjWo45W9KApHslHdUSn11iA5LOaokfLOkWSfdJurxcSUVERJfUGf21WNLElu1JkhbVOPezwKdtvw6YBZwmaehJ/AttzyzLinLeQ4ATgNcDs4GvSZogaQLwVeBoqif5T2w5zxfKuWYAjwKn1MgrIiI6pM7trz+x/djQhu1HgTePdpDtDbZvL+tPUvXDTBnhkDnAUttP2/4FMAAcWpYB2/fbfgZYCsyRJOAI4Ipy/GJgbo3fExERHVKnqOwhadLQhqT92MFpWiRNpypEt5TQ6ZLulLSo5dxTgNZRZoMltr34K4HHbD87LB4REV1Sp6h8CfiJpPMknQf8BPiHul8g6aXAt4EzbT8BXAy8mqrDf0M5P4DaHO4xxNvlMF/SGklrNm3aVDf1iIjYQXU66pcAxwEPAxuBY21/o87Jy3tXvg180/aV5XwP295i+zngX6hub0F1pTGt5fCpwEMjxB8BJkrac1i83W9YaLvfdn9fX1+d1CMiYgzqvvnxZ8CVwFXAU5IOGu2A0udxCbDe9pdb4pNbmn0AuLusLwdOkLSPpIOBGcCtwGpgRhnptTdVZ/5y26aa2PK4cvy8kl9ERHTJqH0jkj4BLKC6UtlCddvJwJ+McujhwEeAuyStLbHPUY3emlnO8QDwMYAyA/Iy4B6qkWOn2d5ScjgdWAlMABbZXlfO91lgqaTzgZ9SFbGIiOiSOh3uZwB/XOfZlFa2b6B9v8eKEY65ALigTXxFu+PKjMmHDo9HRER31Ln99SDweKcTiYiI3lfnSuV+4DpJPwCeHgq29pNERERAvaLyy7LsXZaIiIi26rz58fMAkl5i+zedTykiInpV3vwYERGNyZsfIyKiMXnzY0RENCZvfoyIiMbkzY8REdGYEa9UyguyPmL7wzspn4iI6GEjXqmUubfm7KRcIiKix9XpU7lR0j8DlwO/f05l6K2OERERQ+oUlXeUz3NbYqZ6lW9ERMTvjdansgdwse1lOymfiIjoYaP1qTwHnL6TcomIiB5XZ0jxKkn/RdI0SfsNLR3PLCIiek6dPpW/Kp+tz6YYeFXz6URERC+rM0vxwTsjkYiI6H113lF/Uru47SXNpxMREb2szu2vt7Wsvwh4D3A7kKISERFbqXP76xOt25JeAXyjYxlFRETPqjX1/TC/BWaM1qiMFrtW0npJ6ySdUeL7SVol6b7yOanEJekiSQOS7pT0lpZzzSvt75M0ryX+Vkl3lWMukqQx/J6IiGhInTc/fk/S8rJ8H7gXuKrGuZ8FPm37dcAs4DRJhwBnAdfYngFcU7YBjqYqVjOA+cDF5fv3AxYAhwGHAguGClFpM7/luNk18oqIiA6p06fyjy3rzwL/YXtwtINsbwA2lPUnJa2nmj5/DvCu0mwxcB3w2RJfYtvAzZImSppc2q6yvRlA0ipgtqTrgJfbvqnElwBzgatr/KaIiOiAOkXll8AG278DkLSvpOm2H6j7JZKmA28GbgEOLAUH2xskHVCaTQFa3zA5WGIjxQfbxCMiokvq9Kl8C3iuZXtLidUi6aXAt4EzbT8xUtM2MY8h3i6H+ZLWSFqzadOm0VKOiIgxqlNU9rT9zNBGWd+7zskl7UVVUL5p+8oSfrjc1qJ8bizxQWBay+FTgYdGiU9tE9+G7YW2+2339/X11Uk9IiLGoE5R2STp/UMbkuYAj4x2UBmJdQmw3vaXW3YtB4ZGcM3j+U7/5cBJZRTYLODxcptsJXCkpEmlg/5IYGXZ96SkWeW7TqLeAIKIiOiQOn0qpwLfLC/qguoKoe1T9sMcDnwEuEvS2hL7HPD3wDJJp1D11xxf9q0AjgEGqIYtnwxge7Ok84DVpd25Q532wMeBS4F9qTro00kfEdFFdR5+/Dkwq/SNyPaTdU5s+wba93tA9VT+8PZm60krW/ctAha1ia8B3lAnn4iI6Lw6z6n8raSJtp8qQ4MnSTp/ZyQXERG9pU6fytG2HxvasP0o1W2qiIiIrdQpKhMk7TO0IWlfYJ8R2kdExDhVp6P+fwHXSPo61XMgf0X1JHxERMRW6nTU/4OkO4H3ltB5tld2Nq2IiOhFda5UAH4K7EV1pfLTzqUTERG9rM7orw8CtwLHAR8EbpF0XKcTi4iI3lPnSuW/Am+zvRFAUh/wb8AVnUwsIiJ6T53RX3sMFZTi1zWPi4iIcabOlcoPJa0ELivbH6KaUiUiImIrdUZ/fUbSscCfUk27stD2dzqeWURE9Jxao7/KtPVXjtowIiLGtfSNREREY1JUIiKiMdstKpKuKZ9f2HnpRERELxupT2WypD8D3i9pKcPejWL79o5mFhERPWekonIOcBbVu9+/PGyfgSM6lVRERPSm7RYV21cAV0j6b7bP24k5RUREj6rznMp5kt4PvLOErrP9/c6mFRERvajOhJJ/B5wB3FOWM0osIiJiK3UefnwfMNP2cwCSFlNNf392JxOLiIjeU/c5lYkt66/oRCIREdH76hSVvwN+KunScpVyG/C3ox0kaZGkjZLubon9jaRfSVpblmNa9p0taUDSvZKOaonPLrEBSWe1xA+WdIuk+yRdLmnvuj86IiI6Y9SiYvsyYBbV3F9XAm+3vbTGuS8FZreJX2h7ZllWAEg6BDgBeH055muSJkiaAHwVOBo4BDixtAX4QjnXDOBR4JQaOUVERAfVuv1le4Pt5bavsv1/ax5zPbC5Zh5zgKW2n7b9C2AAOLQsA7bvt/0MsBSYI0lUz8kMvShsMTC35ndFRESHdGPur9Ml3Vluj00qsSnAgy1tBktse/FXAo/ZfnZYvC1J8yWtkbRm06ZNTf2OiIgYZmcXlYuBVwMzgQ3Al0pcbdp6DPG2bC+03W+7v6+vb8cyjoiI2kYsKpL2aO1of6FsP2x7Sxme/C9Ut7egutKY1tJ0KvDQCPFHgImS9hwWj4iILhqxqJT/+N8h6aAmvkzS5JbNDwBDBWs5cIKkfSQdDMwAbgVWAzPKSK+9qTrzl9s2cC1wXDl+HnBVEzlGRMTY1Xn4cTKwTtKtwG+GgrbfP9JBki4D3gXsL2kQWAC8S9JMqltVDwAfK+daJ2kZ1RP7zwKn2d5SznM6sBKYACyyva58xWeBpZLOp3oY85I6PzgiIjqnTlH5/FhObPvENuHt/off9gXABW3iK4AVbeL38/zts4iI2AXUmVDyx5L+EJhh+98kvZjqqiEiImIrdSaU/M9Uz4P8zxKaAny3k0lFRERvqjOk+DTgcOAJANv3AQd0MqmIiOhNdYrK0+VpdgDKMN7tPhMSERHjV52i8mNJnwP2lfTnwLeA73U2rYiI6EV1ispZwCbgLqohwCuAv+5kUhER0ZvqjP56rkx5fwvVba97y8OHERERWxm1qEh6H/A/gJ9Tzbl1sKSP2b6608lFRERvqfPw45eAd9seAJD0auAHQIpKRERspU6fysahglLcD2zsUD4REdHDtnulIunYsrpO0gpgGVWfyvFUEz1GRERsZaTbX3/Rsv4w8GdlfRMwadvmEREx3m23qNg+eWcmEhERva/O6K+DgU8A01vbjzb1fUREjD91Rn99l2rK+u8Bz3U2nYiI6GV1isrvbF/U8UwiIqLn1SkqX5G0APgR8PRQ0PbtHcsqIiJ6Up2i8kbgI8ARPH/7y2U7Ypf0y3Pf2O0UxoWDzrmr2ynELqZOUfkA8KrW6e8jIiLaqfNE/R3AxE4nEhERva/OlcqBwM8krWbrPpUMKY6IiK3UKSoLxnJiSYuA/0Q1d9gbSmw/4HKqZ14eAD5o+1FJAr4CHAP8Fvjo0EAASfN4/v0t59teXOJvBS4F9qV6x8sZmZI/IqK7Rr39ZfvH7ZYa574UmD0sdhZwje0ZwDVlG+BoYEZZ5gMXw++L0ALgMOBQYIGkoSliLi5th44b/l0REbGTjVpUJD0p6Ymy/E7SFklPjHac7euBzcPCc4DFZX0xMLclvsSVm4GJkiYDRwGrbG+2/SiwCphd9r3c9k3l6mRJy7kiIqJL6rz58WWt25LmUl01jMWBtjeU826QdECJTwEebGk3WGIjxQfbxNuSNJ/qqoaDDjpojKlHRMRo6oz+2ort79L8Mypq91VjiLdle6Htftv9fX19Y0wxIiJGU2dCyWNbNvcA+hnhP+CjeFjS5HKVMpnnX/Y1CExraTcVeKjE3zUsfl2JT23TPiIiuqjOlcpftCxHAU9S9YGMxXJgXlmfB1zVEj9JlVnA4+U22UrgSEmTSgf9kcDKsu9JSbPKyLGTWs4VERFdUqdPZUzvVZF0GdVVxv6SBqlGcf09sEzSKcAvqd4iCdWQ4GOAAaohxSeX794s6Tyef9PkubaHOv8/zvNDiq8uS0REdNFIrxM+Z4TjbPu8kU5s+8Tt7HpPu5MBp23nPIuARW3ia4A3jJRDRETsXCNdqfymTewlwCnAK4ERi0pERIw/I71O+EtD65JeBpxBdVtqKfCl7R0XERHj14h9KuWJ9k8BH6Z6WPEt5SHEiIiIbYzUp/JF4FhgIfBG20/ttKwiIqInjTSk+NPAH1BN5vhQy1QtT9aZpiUiIsafkfpUdvhp+4iIGN9SOCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGdKWoSHpA0l2S1kpaU2L7SVol6b7yOanEJekiSQOS7pT0lpbzzCvt75M0rxu/JSIintfNK5V3255pu79snwVcY3sGcE3ZBjgamFGW+cDFUBUhYAFwGHAosGCoEEVERHfsSre/5gCLy/piYG5LfIkrNwMTJU0GjgJW2d5s+1FgFTB7ZycdERHP61ZRMfAjSbdJml9iB9reAFA+DyjxKcCDLccOltj24hER0SXbfUd9hx1u+yFJBwCrJP1shLZqE/MI8W1PUBWu+QAHHXTQjuYaERE1deVKxfZD5XMj8B2qPpGHy20tyufG0nwQmNZy+FTgoRHi7b5voe1+2/19fX1N/pSIiGix04uKpJdIetnQOnAkcDewHBgawTUPuKqsLwdOKqPAZgGPl9tjK4EjJU0qHfRHllhERHRJN25/HQh8R9LQ9/+r7R9KWg0sk3QK8Evg+NJ+BXAMMAD8FjgZwPZmSecBq0u7c21v3nk/IyIihtvpRcX2/cCb2sR/DbynTdzAads51yJgUdM5RkTE2OxKQ4ojIqLHpahERERjujWkuCe89TNLup3Cbu+2L57U7RQiokG5UomIiMakqERERGNSVCIiojEpKhER0ZgUlYiIaEyKSkRENCZFJSIiGpOiEhERjUlRiYiIxqSoREREY1JUIiKiMSkqERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGNSVGJiIjGpKhERERjer6oSJot6V5JA5LO6nY+ERHjWU8XFUkTgK8CRwOHACdKOqS7WUVEjF89XVSAQ4EB2/fbfgZYCszpck4REeNWrxeVKcCDLduDJRYREV2wZ7cTeIHUJuZtGknzgfll8ylJ93Y0q+7aH3ik20nUpX+c1+0UdiU99bcDYEG7fwXHrZ76++mTO/y3+8M6jXq9qAwC01q2pwIPDW9keyGwcGcl1U2S1tju73YesePyt+tt+ftVev3212pghqSDJe0NnAAs73JOERHjVk9fqdh+VtLpwEpgArDI9roupxURMW71dFEBsL0CWNHtPHYh4+I2324qf7velr8fIHubfu2IiIgx6fU+lYiI2IWkqOwmMl1N75K0SNJGSXd3O5fYMZKmSbpW0npJ6ySd0e2cui23v3YDZbqa/wP8OdUw69XAibbv6WpiUYukdwJPAUtsv6Hb+UR9kiYDk23fLullwG3A3PH8716uVHYPma6mh9m+Htjc7Txix9neYPv2sv4ksJ5xPqtHisruIdPVRHSZpOnAm4FbuptJd6Wo7B5qTVcTEZ0h6aXAt4EzbT/R7Xy6KUVl91BrupqIaJ6kvagKyjdtX9ntfLotRWX3kOlqIrpAkoBLgPW2v9ztfHYFKSq7AdvPAkPT1awHlmW6mt4h6TLgJuCPJQ1KOqXbOUVthwMfAY6QtLYsx3Q7qW7KkOKIiGhMrlQiIqIxKSoREdGYFJWIiGhMikpERDQmRSUiIhqTohIREY1JUYkAJP2kRpszJb24w3nMHO05B0kflfTPDX9v4+eM8SlFJQKw/Y4azc4EdqiolNcS7IiZwLh+eC56W4pKBCDpqfL5LknXSbpC0s8kfVOVTwJ/AFwr6drS9khJN0m6XdK3yqSCSHpA0jmSbgCOl/RqST+UdJuk/y3ptaXd8ZLulnSHpOvLFDvnAh8qT2Z/qEbefZK+LWl1WQ6XtEfJYWJLuwFJB7Zr3/g/zBjX9ux2AhG7oDcDr6ealPNG4HDbF0n6FPBu249I2h/4a+C9tn8j6bPAp6iKAsDvbP8pgKRrgFNt3yfpMOBrwBHAOcBRtn8laaLtZySdA/TbPr1mrl8BLrR9g6SDgJW2XyfpKuADwNfLdz5g+2FJ/zq8PfC6F/jPK+L3UlQitnWr7UEASWuB6cANw9rMAg4BbqzmFGRvqvm7hlxejn8p8A7gW6UdwD7l80bgUknLgLHObvte4JCWc7+8vIHwcqqi9XWqCUYvH6V9RCNSVCK29XTL+hba/3siYJXtE7dzjt+Uzz2Ax2zPHN7A9qnlKuJ9wFpJ27SpYQ/g7bb/31bJSTcBr5HUB8wFzh+l/Ri+OmJb6VOJqO9JYOj/6m8GDpf0GgBJL5b0R8MPKC9s+oWk40s7SXpTWX+17VtsnwM8QvVOnNbvqONHVDNUU845s3yvge8AX6aalv3XI7WPaEqKSkR9C4GrJV1rexPwUeAySXdSFZnXbue4DwOnSLoDWAfMKfEvSrpL0t3A9cAdwLVUt6dqddQDnwT6Jd0p6R7g1JZ9lwN/yfO3vkZrH/GCZer7iIhoTK5UIiKiMemoj9hFSToZOGNY+Ebbp3Ujn4g6cvsrIiIak9tfERHRmBSViIhoTIpKREQ0JkUlIiIak6ISERGN+f+UQGoL+CmGNwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.countplot(train.interest_level);\n",
    "pyplot.xlabel('interest_level');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "总共有三类样本，每类样本分布的并不均匀，一般类别之间的数量差别在十倍以上时才考虑使用class_weight，这里暂不考虑"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['interest_level']\n",
    "\n",
    "train = train.drop([\"interest_level\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值（337），除了需要调优的两个参数，其余参数继续默认值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [5, 6, 7], 'min_child_weight': [2, 3, 4]}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = [5,6,7]\n",
    "min_child_weight = [2,3,4]\n",
    "param_test2_2 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上一轮粗调得到的max_depth = 6，min_child_weight = 3，这轮进行细调，分别以6和3为中心值，两边各取一个值进行调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_search.py:761: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -0.58383, std: 0.00349, params: {'max_depth': 5, 'min_child_weight': 2},\n",
       "  mean: -0.58390, std: 0.00348, params: {'max_depth': 5, 'min_child_weight': 3},\n",
       "  mean: -0.58399, std: 0.00354, params: {'max_depth': 5, 'min_child_weight': 4},\n",
       "  mean: -0.58467, std: 0.00367, params: {'max_depth': 6, 'min_child_weight': 2},\n",
       "  mean: -0.58410, std: 0.00377, params: {'max_depth': 6, 'min_child_weight': 3},\n",
       "  mean: -0.58407, std: 0.00315, params: {'max_depth': 6, 'min_child_weight': 4},\n",
       "  mean: -0.58964, std: 0.00403, params: {'max_depth': 7, 'min_child_weight': 2},\n",
       "  mean: -0.58724, std: 0.00436, params: {'max_depth': 7, 'min_child_weight': 3},\n",
       "  mean: -0.58815, std: 0.00382, params: {'max_depth': 7, 'min_child_weight': 4}],\n",
       " {'max_depth': 5, 'min_child_weight': 2},\n",
       " -0.5838311336148692)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_2 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=337,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.5,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_2 = GridSearchCV(xgb2_2, param_grid = param_test2_2, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_2.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_2.grid_scores_, gsearch2_2.best_params_,     gsearch2_2.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([651.88127642, 633.28183527, 650.19149656, 783.21042681,\n",
       "        788.76142802, 770.92543154, 817.02817054, 801.80903134,\n",
       "        738.40389476]),\n",
       " 'mean_score_time': array([2.15684061, 1.61587973, 1.81501288, 2.62235212, 2.43682852,\n",
       "        2.63356047, 3.22815719, 3.13949761, 2.42321787]),\n",
       " 'mean_test_score': array([-0.58383113, -0.58390255, -0.58398572, -0.58467475, -0.58409798,\n",
       "        -0.58406584, -0.58963771, -0.58724368, -0.58814807]),\n",
       " 'mean_train_score': array([-0.47691834, -0.47985162, -0.48250501, -0.41670864, -0.42294951,\n",
       "        -0.42881501, -0.34601727, -0.35831669, -0.36637739]),\n",
       " 'param_max_depth': masked_array(data=[5, 5, 5, 6, 6, 6, 7, 7, 7],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'param_min_child_weight': masked_array(data=[2, 3, 4, 2, 3, 4, 2, 3, 4],\n",
       "              mask=[False, False, False, False, False, False, False, False,\n",
       "                    False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'max_depth': 5, 'min_child_weight': 2},\n",
       "  {'max_depth': 5, 'min_child_weight': 3},\n",
       "  {'max_depth': 5, 'min_child_weight': 4},\n",
       "  {'max_depth': 6, 'min_child_weight': 2},\n",
       "  {'max_depth': 6, 'min_child_weight': 3},\n",
       "  {'max_depth': 6, 'min_child_weight': 4},\n",
       "  {'max_depth': 7, 'min_child_weight': 2},\n",
       "  {'max_depth': 7, 'min_child_weight': 3},\n",
       "  {'max_depth': 7, 'min_child_weight': 4}],\n",
       " 'rank_test_score': array([1, 2, 3, 6, 5, 4, 9, 7, 8]),\n",
       " 'split0_test_score': array([-0.57837319, -0.57819605, -0.57801336, -0.57828958, -0.57821863,\n",
       "        -0.57829757, -0.58258619, -0.5794013 , -0.58135505]),\n",
       " 'split0_train_score': array([-0.47749025, -0.48030373, -0.48375844, -0.41786701, -0.42405239,\n",
       "        -0.42933091, -0.34851445, -0.36146617, -0.36956211]),\n",
       " 'split1_test_score': array([-0.58299215, -0.58338811, -0.58311238, -0.58303987, -0.58312449,\n",
       "        -0.58377279, -0.5876967 , -0.5878283 , -0.58820162]),\n",
       " 'split1_train_score': array([-0.47822321, -0.4810964 , -0.48371788, -0.41886659, -0.42526734,\n",
       "        -0.43132266, -0.34529829, -0.35759164, -0.3650784 ]),\n",
       " 'split2_test_score': array([-0.58271703, -0.58304238, -0.58432007, -0.58656538, -0.58277766,\n",
       "        -0.58449794, -0.59198112, -0.58791383, -0.58999795]),\n",
       " 'split2_train_score': array([-0.47608596, -0.47953354, -0.48193904, -0.41642383, -0.4239282 ,\n",
       "        -0.42886192, -0.34543868, -0.35866488, -0.36684632]),\n",
       " 'split3_test_score': array([-0.58662913, -0.58644602, -0.58568632, -0.58690547, -0.58777424,\n",
       "        -0.58658534, -0.5930361 , -0.58818733, -0.58819972]),\n",
       " 'split3_train_score': array([-0.47768698, -0.47993413, -0.48278957, -0.4155262 , -0.4213768 ,\n",
       "        -0.42759794, -0.34506984, -0.35641813, -0.36538971]),\n",
       " 'split4_test_score': array([-0.58844556, -0.58844159, -0.58879793, -0.58857461, -0.58859626,\n",
       "        -0.5871765 , -0.59288944, -0.59288933, -0.5929875 ]),\n",
       " 'split4_train_score': array([-0.47510528, -0.47839028, -0.48032009, -0.41485955, -0.4201228 ,\n",
       "        -0.42696161, -0.3457651 , -0.35744264, -0.36501042]),\n",
       " 'std_fit_time': array([15.14474194,  6.34823826,  8.66373061,  4.93025564,  6.42540099,\n",
       "        33.87885513, 13.09221442,  3.09258742, 87.6919772 ]),\n",
       " 'std_score_time': array([0.60140515, 0.15972029, 0.26248449, 0.39138036, 0.38947725,\n",
       "        0.45747638, 0.09563985, 0.2688247 , 0.42262092]),\n",
       " 'std_test_score': array([0.00348864, 0.00348248, 0.00353807, 0.00366612, 0.00376843,\n",
       "        0.00314864, 0.00402759, 0.00436013, 0.00382074]),\n",
       " 'std_train_score': array([0.00114897, 0.00089413, 0.00128198, 0.00147628, 0.00189791,\n",
       "        0.00151479, 0.00126876, 0.00172827, 0.00172591])}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_2.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.583831 using {'max_depth': 5, 'min_child_weight': 2}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAELCAYAAADz6wBxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3Xl4FeX58PHvnQUi+y5LgCCyCUQ2EaEsLigIYiNaweUVLNKyCGhr0ba/Ktafor6vRBBp3a0bCBUbLMqiIFWwLDEsSdhFiOxBtrBlud8/Zk44CVlOcnJystyf6zpXzsw8M+c+k2TumeeZeR5RVYwxxpjiCgl2AMYYY8o3SyTGGGP8YonEGGOMXyyRGGOM8YslEmOMMX6xRGKMMcYvlkiMMcb4xRKJMcYYv1giMcYY45ewYAdQGho0aKBRUVHBDsMYY8qNBg0asGTJkiWqOqiwspUikURFRbF+/fpgh2GMMeWKiDTwpZxVbRljjPGLJRJjjDF+sURijDHGL5WijcQY47v09HRSUlI4d+5csEMxpSQiIoLIyEjCw8OLtb4lEmNMDikpKdSsWZOoqChEJNjhmABTVVJTU0lJSaFVq1bF2oZVbRljcjh37hz169e3JFJJiAj169f36wrUEokx5hKWRCoXf3/flkgK8O/d/2bL0S3BDsMYY8o0ayPJR2ZWJrMTZrPv1D4GthzIpK6TiKodFeywjDGmzLErknyEhoQy/7b5jLt6HN/89A2//NcvmbZmGofPHA52aMZUaMePH+fVV18t1rqxsbGcOXOmhCMqGQMGDCh2DxuffvopSUlJRd5WVFQUnTt3pkuXLvTo0aNYn+0LSyQFqB5enfFdxrP4jsX8qt2v+HTnpwz5ZAixG2I5eeFksMMzpkKqqInEH7kTSVGsWLGChISEgHYTZVVbPmhwWQP+eO0fuf+q+3nl+1d4c8ubzN8+nzGdxzCy/UgiwiKCHaIxATFtUSJJ+0v2pOmqprV48raO+S5//PHH2bVrF126dGHgwIE0atSIjz/+mPPnzxMTE8O0adNIS0vjV7/6FSkpKWRmZvI///M/HDp0iP3793P99dfToEEDVqxYkef2a9SowYQJE1i+fDl169bl2Wef5Q9/+AN79+4lNjaWYcOGsWfPHu6//37S0tIAeOWVV+jduzcLFy5k9uzZLFu2jIMHD9K/f39WrVpF48aNL/mcs2fPMnr0aJKSkujQoQNnz57NXrZ06VKefPJJzp8/T+vWrXn77bepUaMGUVFR3H333dmxf/jhhxw+fJi4uDi+/vprnnnmGf75z38CMH/+fMaPH8/x48d588036du3b7F/J/6yK5IiaF6zOc/3e575t82nc8POvLThJYYuHMonOz4hIysj2OEZUyFMnz6d1q1bk5CQwMCBA9mxYwdr164lISGBDRs2sGrVKr744guaNm3Kxo0b2bJlC4MGDWLSpEk0bdqUFStW5JtEANLS0hgwYAAbNmygZs2a/PnPf2bZsmUsXLiQv/zlLwA0atSIZcuWER8fz7x585g0aRIAMTExNG7cmNmzZ/PQQw8xbdq0PJMIwJw5c6hWrRqbNm3iT3/6Exs2bADg6NGjPPPMMyxfvpz4+Hh69OjBSy+9lL1erVq1WLt2LRMnTmTKlCn07t2bYcOG8eKLL5KQkEDr1q0ByMjIYO3atcTGxjJt2jQA9u/fz6233pq9LRHh5ptvpnv37rz22mt+/FYKZlckxdC+Xnv+dtPfWHdwHTM2zODJ1U/ybuK7TOo6iRta3GC3TpoKo6Arh9KwdOlSli5dSteuXQE4ffo0O3bsoG/fvvz+979n6tSpDB06tEhn41WqVGHQIKdn9M6dO1O1alXCw8Pp3Lkze/bsAZyn+ydOnEhCQgKhoaFs3749e/1Zs2bRqVMnevXqxciRI/P9nFWrVmUnoOjoaKKjowH47rvvSEpKok+fPgBcuHCB6667Lns9zzZHjhzJI488ku/277jjDgC6d++eHXfTpk1ZvHhxdplvv/2Wpk2bcvjwYQYOHEj79u3p16+fT/upKCyR+OGaxtfwwa0f8OXeL3k5/mWmrJxCdMNopnSbwjWNrwl2eMaUe6rKE088wW9+85tLlm3YsIHFixfzxBNPcPPNN2dfTRQmPDw8+2QvJCSEqlWrZr/PyHBqFmbMmMHll1/Oxo0bycrKIiLiYvX1Tz/9REhICIcOHSIrK4uQkPwrdvI6qVRVBg4cyEcffVToOgWdlHriDg0NzY47t6ZNmwLOFVZMTAxr164NSCKxqi0/iQg3tbyJhbcv5KnrnuJg2kEeXPIg45ePZ9uxbcEOz5hyp2bNmpw6dQqAW265hbfeeovTp08DzkH88OHD7N+/n2rVqnHffffx+9//nvj4+EvW9ceJEydo0qQJISEhvPfee2RmZgJOddLo0aP58MMP6dChQ44qqdz69evHBx98AMCWLVvYtGkTAL169eLbb79l586dAJw5cybHFc+8efOyf3quVIrzvdLS0rLXSUtLY+nSpXTq1KlI2/CVXZGUkLCQMIa3Hc6QK4bw4dYPeWPzG9y16C6GXDGECV0mEFkzMtghGlMu1K9fnz59+tCpUycGDx7MPffck31ArVGjBu+//z47d+7kscceIyQkhPDwcObMmQPA2LFjGTx4ME2aNCmwnaQw48ePZ/jw4cyfP5/rr7+e6tWrA/Dss8/St29f+vbtS5cuXbjmmmsYMmQIHTp0uGQb48aNY/To0URHR9OlSxd69uwJQMOGDXnnnXcYOXIk58+fB+CZZ56hbdu2AJw/f55rr72WrKys7KuWESNG8NBDDzFz5kwWLFiQb9z79+9nzJgxLF68mEOHDhETEwM4CfCee+7JrtIraaKqAdlwWdKjRw8t7REST5w/wZtb3uTD5A/J1Ezubnc3Y6PHUi+iXqnGYUxRJScn53lgNIHnGc21QQOfBiYsUXn93kVkg6oW+gCKVW0FSO2qtXm0+6N8FvMZt7e+nY+2fsTgfw5mTsIc0tLTgh2eMcaUGKvaCrDG1RvzVO+n+D8d/w+z4mfx6sZXmbttLmOjx/Krtr8iPLR4/f8bYwp27bXXZlcdebz33nt07ty5RD9nyZIlTJ06Nce8Vq1asXDhwiJvy3P3VXljVVulbNORTcTGx7Lu4Doia0QysetEBrcaTIjYxaEpG6xqq3Kyqq1yJLphNG/e/CZzbppD9fDqPP6fx7n7s7v55qdvqAxJ3RhT8VgiCQIR4RfNfsHHt33M9L7TOXXhFOOWj+PXS3/NpiObgh2eMcYUiSWSIAqREIZcMYRFv1zE4z0fZ9fxXdy7+F4eWfEIu0/sDnZ4xhjjE0skZUB4aDj3driXxXcsZvzV41m9fzV3/OsOnlr9FIfSDgU7PGOMKZAlkjKkenh1xnUZx+I7FjOi/Qj+tetfDFk4hBkbZnDi/Ilgh2dMqaio3cgHYzyS48ePc+edd9K+fXs6dOjAmjVrivX5hQloIhGRQSKyTUR2isjjeSwfJSJHRCTBfY3xWvaCiCSKSLKIzBS30xkRuVtENrnLXghk/MFS/7L6PN7zcRb9chEDWw7k7S1vM/iTwby15S3OZZwLdnjGBFRFTST+KO54JJMnT2bQoEFs3bqVjRs3BuxuvIA9RyIiocBsYCCQAqwTkThVzb035qnqxFzr9gb6ANHurG+A/iKyGXgR6K6qR0TkXRG5UVW/DNT3CKbImpE81/c5RnUcRWx8LDM2zOCD5A8Yf/V4br/ydsJC7DEgE2CfPw4HN5fsNht3hsHT811s45GUzHgkJ0+eZNWqVbzzzjuA0+txlSpVfP89FUEgr0h6AjtVdbeqXgDmArf7uK4CEUAVoCoQDhwCrgC2q+oRt9xyYHiJRl0GtavXjjk3zeGtW96icbXGPLXmKe6Iu4PlPy63W4ZNhWPjkZTMeCS7d++mYcOGjB49mq5duzJmzJjsxFjSAnlK2wzY5zWdAlybR7nhItIP2A48oqr7VHWNiKwADgACvKKqySJSF2gvIlHu9n6Jk2wqhWsaX8P7t77PV3u/4uXvX+aRlY8Q3SCaKd2t23oTIAVcOZQGG4+k+OORZGRkEB8fz6xZs7j22muZPHky06dP569//avP+8pXgUwkeXWkn/v0eRHwkaqeF5HfAu8CN4jIlUAHwNNl7jIR6aeqq0RkHDAPyAJW41ylXPrhImOBsQAtWrTw+8uUFSLCjS1vpH/z/sTtimN2wmweXPIgfZr14ZFuj9CuXrtgh2hMibHxSIo/HklkZCSRkZFce61z/n7nnXcyfXpgTgwCWbWVAjT3mo4E9nsXUNVUVfV0hvM60N19HwN8p6qnVfU08DnQy11nkapeq6rXAduAHXl9uKq+pqo9VLVHw4YNS+xLlRVhIWHc0eYO/h3zbx7t/iibj2zmrkV38fh/HiflVEqwwzOm2Gw8kpIZj6Rx48Y0b96cbduccZG+/PJLrrrqqiJtw1eBTCTrgDYi0kpEqgAjgDjvAiLSxGtyGJDsvt+L07geJiLhQH/PMhFp5P6sC4wH3gjgdyjzIsIiGN1pNIvvWMyDnR5k+Y/Lue3T23juv8+RejY12OEZU2Te45EsW7YsezySzp07c+edd3Lq1Ck2b95Mz5496dKlC//7v//Ln//8Z+DieCTXX3+9XzGMHz+ed999l169erF9+/Y8xyN56aWXeOONN0hOTs5zG+PGjeP06dNER0fzwgsv5DkeSXR0NL169WLr1q3Z63nGI3n55ZeZMWMG4IxH8uKLL9K1a1d27dqVb9y5x2yfNWsW9957L9HR0SQkJPDHP/7Rr/2Sn4B22igitwKxQCjwlqr+r4g8DaxX1TgReQ4ngWQAx4BxqrrVvePrVaAfTnXYF6r6qLvNj4Cr3Y94WlXnFhZHWeq0MdAOpR1izsY5fLrzU6qGVuWBjg/wQMcHqB5ePdihmXLCOm0MnvI6Hon1/ltB/XDiB2Z9P4tlPy6jXkQ9xkaP5a62d1EltNLcm2CKyRJJ8JTXRGIPIlRQrWq34qUBL7H5yGZi42OZvnY67yW9x8SuE7m11a3Wbb2p8Gw8ktJjVySVgKqyev9qYuNj2XpsK+3qtmNyt8n8otkvCrwrxFROdkVSOdl4JKZAIkKfZn2YN3Qez/d9nrT0NMZ/OZ4HlzzIxiMbgx2eMaacs0RSiYRICLdecStxv4zjiZ5PsPvEbu5bfB9TVkyxbuuNMcVmiaQSCg8N554O9/D5HZ8zvst41uxfQ8y/Ynhy9ZMcTDsY7PCMMeWMJZJKrFp4NcZdPY7Ph3/OPe3vIW5XHEMXDuWl9S9Zt/UmaCpq77+l3Y38tm3b6NKlS/arVq1axMbGFuvzC2OJxFAvoh5Te07ls5jPuLnlzbyT+A6DPxnMm5vf5GzG2cI3YEwJqqiJxB/F6Ua+Xbt2JCQkZHd2Wa1aNWJiYgISnyUSk61ZjWY82/dZ5t82n66NuhIbH8vQT4ayYPsCMrIu7cvHmEDw7kb+scce48UXX+Saa64hOjqaJ598EnB68B0yZAhXX301nTp1Yt68ecycOTO7G/mCnmyvUaMGU6dOpXv37tx0002sXbuWAQMGcMUVVxAX53S+sWfPHvr27Uu3bt3o1q0bq1evBmDhwoXcdNNNqCoHDhygbdu2HDyYd3Xw2bNnGTFiBNHR0dx9992XdCN/3XXX0a1bN+66667sLmCioqKYOnUqPXv2pGfPnuzcuZPVq1cTFxfHY489RpcuXbKfbJ8/fz49e/akbdu2/Oc//ylwn3755Ze0bt2ali1b+vhbKBp7jsRcol29dsy+cTbrD65nRvwMpq2ZxruJ7zKp2yRuanGT3TJciTy/9nm2HttaeMEiaF+vPVN7Ts13+fTp09myZQsJCQksXbqUBQsWsHbtWlSVYcOGsWrVKo4cOULTpk3597//DTh9Y9WuXZuXXnqJFStWFPhAn6cb+eeff56YmJjsbuSTkpJ44IEHGDZsWHY38hEREezYsYORI0eyfv16YmJi+Oc//8ns2bP54osvfO5GftOmTXTr1g3I2Y189erVef7553nppZeyO530dCP/j3/8gylTpvDZZ58xbNgwhg4dyp133pm9fU838osXL2batGksX76c/fv3M2bMmOwegD3mzp1bYE/F/rJEYvLVo3EP3h/8Pl/t+4qZ8TN5dOWjdG7QmSndptCzSc9gh2cqAetGvvjdyHtcuHCBuLg4nnvuuUL3TXFZIjEFEhFubHEj/SP7s2jXImYnzObXS39Nn6Z9mNJ9Cu3rtQ92iCaACrpyKA3WjXzxu5H3+Pzzz+nWrRuXX355vmX8ZW0kxidhIWHEtInhs5jP+F3337H5qNNt/dRVU9l3al/hGzDGR9aNfMl0I+/x0UcfBbRaCyyRmCKKCItgVKdRfD78c8Z0HsNXe79i2KfDePa/z3L07NFgh2cqAOtGvuS6kT9z5gzLli3LrgYLFOtry/jl8JnDzNk4h4U7FlIltIrTbf1VD1CjSo1gh2aKyfraCp7y2vuvXZEYvzSq1ognr3uShbcv5BfNfsHfNv6NWz+5lfeT3udC5oVgh2eMKQXW2G5KhKfb+i1HtxC7IZbn1z3P+8nvM6HLBG5tdSuhIaHBDtFUMtaNfOmxqi1T4lSVNfvXEBsfS/KxZNrWbcvkbpPp26yvPYNSDljVVuVkVVumTBERejfrzdyhc3mh3wuczTjLhC8nMHrJaOu2vpyoDCeY5iJ/f9+WSEzAhEgIg1sN5l+3/4s/XvtHfjjxA/ctvo/JX01m93Hrtr6sioiIIDU11ZJJJaGqpKam5nhWpqisasuUmjPpZ/hH0j94J/Edzmac5fbWtzO+y3gaV8+7iwkTHOnp6aSkpHDu3Llgh2JKSUREBJGRkYSHh+eY72vVliUSU+qOnTvG65teZ962eQjCPR3uYUznMdSuWjvYoRljvFgi8WKJpGz66fRPvJrwKot2LaJGeA0e7Pwg93a4l8vCLgt2aMYYLJHkYImkbNv+83Zmxs/k65SvaXRZI37b5bfEXBlDWIjdnW5MMFki8WKJpHzYcGgDMzbMYOORjUTViuLhrg8zsOVAu2XYGB9laRZ7Tu4h8WgiSalJ/HDiB+bcNKfY/0O+JhI75TNlRvfLu/Pe4PdYsW8FM+Nn8ruvf0en+p2Y0n0K1za5NtjhGVOmZGkW+07tI/FoIompzis5NZkzGc4IkRGhEbSv156TF04GvP0xoFckIjIIeBkIBd5Q1em5lo8CXgR+cme9oqpvuMteAIbg3KK8DJisqioiI4E/AgrsB+5T1QJ7C7QrkvInMyuTuF1xvLrxVQ6mHaR3095M6TaFDvXtQTlT+agqKadTSExNJOlokvMzNYnT6U6vyFVCqtC+Xnuuqn8VHRt05Kr6V3FF7Sv8rh4OetWWiIQC24GBQAqwDhipqkleZUYBPVR1Yq51e+MkmH7urG+AJ9yf+4GrVPWom2zOqOpTBcViiaT8Op95nrlb5/L65tc5cf4Eg6MG83DXh2leq3mwQzMmIFSVA2kHnKsM92ojKTWJkxdOAhAeEk7bum3pWL9jdtJoXac14SHhhWy56MpC1VZPYKeq7nYDmgvcDvgygr0CEUAVQIBw4JD7XoDqIpIK1AJ2lnzopqyoGlqVBzo+QEybGN7Z8g7vJb3Hsh+XMbztcH579W9pcFnp95JqTElRVQ6dOZSdNJJSnauN4+ePAxAmYbSp24aBLQfSsUFHOtbvSJs6bQgPLfmk4Y9AJpJmgPeIRylAXhXdw0WkH87VyyOquk9V14jICuAATuJ4RVWTAURkHLAZSAN2ABMC+B1MGVGrSi0mdZvEiPYj+NvGv7Fg+wLidsVx/1X3M7rjaOu23pQLh88czk4WnquNY+eOARAqobSu05rrm1+ffbXRpm4bqoZWDXLUhQtk1dZdwC2qOsadvh/oqaoPe5WpD5xW1fMi8lvgV6p6g4hcidO2crdbdBkwFVgDfAGMBXYDs4CDqvpMHp8/1i1HixYtuv/4448B+Z4mOPac2MOs72ex9Mel1K1al4eiH+LudndTJbRKsEMzBoCjZ49mJw1Pu8aRs0cAp/ugK2pf4bRpuEmjXd12RIQVv5uSQCgLbSTXAU+p6i3u9BMAqprnCPRum8oxVa0tIo8BEar6V3fZX4BzwApguqre6M7vBzyuqrfmtU0PayOpuBKPJjIjfgb/PfBfmlZvyoSuExjSaoh1W29K1c/nfr7kSuPQmUMACEJU7SgnYXgljWrh1YIcdeHKQhvJOqCNiLTCuStrBHCPdwERaaKqB9zJYYBnzMq9wEMi8hxO1VZ/INbdzlUi0lBVj+A05Oc9zqWpFDo26MgbN7/B6v2rid0Qy5+++RNvb3mbKd2m0C+ynz2DYkrcifMnLl5ppCaReDSR/Wn7s5e3rNWSbpd3y04cHep3oHp49SBGHHgBSySqmiEiE4ElOLf/vqWqiSLyNLBeVeOASSIyDMgAjgGj3NUXADfgtIUo8IWqLgIQkWnAKhFJB370WsdUYr2b9qZXk14s3bOUmd/PZOJXE+nWqBuPdH+ELo26BDs8U06dvHCS5NTkHFcbKadTspdH1oikc8POjGg/go71O9K+fntqVakVxIiDw55sNxVOelY6n2z/hDkb55B6LpUBzQcwuetkrqx7ZbBDM2VYWnoaSalJOa42fjx5sW21afWm2bfbdqzv/KzoHY0GvY2kLLFEUjmdST/De0nv8Xbi25zNOMuw1sOY0GWCdVtvOJN+hq3Htl680khNZM+JPSjO8fDyapdnt2d4kkbdiLpBjrr0WSLxYomkcvv53M+8vvl15m6diyCMbD+SMZ3HUCeiTrBDM6XgbMZZth3bluNKY/eJ3WRpFgANL2voJIsGF6807PkkhyUSL5ZIDMD+0/uZnTCbRbsWUT28Og92crqtLw93zxjfnM88z/Zj23Ncaew6votMzQSgXkS9S640GlVrFOSoyy5LJF4skRhvO37ewcz4maxMWUmDyxow7upxxLSJCUgXEyZw0jPT2X58e/YT4UmpSez4eQcZmgFAnap1spOFJ3FcXu1yu5OvCCyReLFEYvISfyieGRtmkHAkgZa1WjKx60RuaXmLHWjKoPSsdHYd35WjG5HtP28nPSsdcHo+8H64r2P9jjSp3sR+l36yROLFEonJj6qyct9KZn4/k53Hd9KxfkemdJ9Crya9gh1apZWRlcHuE7tzXGlsPbaVC1kXAKgRXuPinVNuu0ZkjUhLGgFgicSLJRJTmMysTBbtXsTshNkcTDvIdU2uY0r3KVxV/6pgh1ahZWZlOgMxeT3ct/XYVs5lngOgWlg1OtTvkOOp8OY1mxMiIUGOvHKwROLFEonxVe5u6wdFDeLhrg/TolaLYIdW7mVpFj+e/DFH0kg+lszZjLMAXBZ2Ge3rtc/RrhFVK8qSRhBZIvFiicQU1akLp3h7y9u8n/w+6Znp1m19EamqM3qfp3v0Y04VVVp6GuAMD9CuXruLVxr1O9KqdivrI62MsUTixRKJKa4jZ47w901/Z8H2BVQJrcJ9He5jdKfR1KxSM9ihlRmqyk+nf8q+3dbTrnHqwinAGYipXd12OW65bV2ntd+j95nAs0TixRKJ8dePJ39k1vezWLJnCXWq1uGhzg9xd/u7y8VYESVJVTmYdjBH0khMTeTE+RMAhIWE0bZu24t3UNXvyJV1rixzAzEZ31gi8WKJxJSUxNREYjfE8t2B72hSvQkTukxg6BVDK2SVjKpy+Mzh7KSRmJpIcmpyjoGY2tRtkyNptKnbxsaEqUAskXixRGJK2pr9a4iNjyUpNYkr61zJ5G6T6R/Zv1zfgnr07NEcY4QnpiZy9OxRwBmIqXWd1lxV7+LDfW3rti1zAzGZkmWJxIslEhMIWZrF0h+XMit+FntP7aVro6480v0RujbqGuzQCpV6NjVHNyJJqUkcPnMYcAZiyh69zytpWFcylU9AEomI1AWaq+omf4IrbZZITCClZ6WzcMdC5mycw9GzRxkQOYBJ3SbRpm6bYIcGwPFzxy9JGgfSDmQvj6oVleOp8A71OljSMEAJJhIRWYkzemEYkAAcAb5W1UdLIM5SYYnElIYz6Wd4P/l93t7yNmnpadnd1jep0aTUYjhx/gTJx5JzVFH9dPqn7OUtarbIkTTa12tvd6CZfJVkIvleVbuKyBicq5EnRWSTqkaXVLCBZonElKbj547z+ubX+WjrRwjCiPYjeKjzQyXebf3pC6dzJI3E1ET2ndqXvbxZjWY5Hu7rUK9DhR+IyZSskkwkm4GbgXeBP6nqOkskxhTuwOkDTrf1uxdRLawaozuN5r4O9xWr2uhM+plLrjT2nNyTvbxJ9SYXk4b708ZbMf4qyURyF/A/wDeqOl5ErgBeVNXhJRNq4FkiMcG08+edvPz9y6zc53Rb/9vo33JH2zvy7bbeMxCT56nwxNREfjjxQ/bofY2qNbokadS/rH5pfiVTSdhdW14skZiy4PvD3xO7IZb4w/G0qNmCh7s+TP/m/dnx844cScN79L76EfXp1KBTjqTRsFrDIH8TU1mU5BXJC8AzwFngC+BqYIqqvl8SgZYGSySmrFBVvk75mpfjX2bn8Z05ltWtWpeODTrmeMCvUbVG5frZFFO++ZpIfOns5mZV/YOIxAApwF3ACqDcJBJjygoRYUDzAfRt1pfFPyzmhxM/ZCeOxtUbW9Iw5ZIvicRTkXsr8JGqHrM/dmP8ExoSym2tbwt2GMaUCF8SySIR2YpTtTVeRBoC5wIbljHGmPKi0BFjVPVx4Dqgh6qmA2nA7YEOzBhjTPlQaCIRkXDgfmCeiCwAfg2k+rJxERkkIttEZKeIPJ7H8lEickREEtzXGK9lL4hIoogki8hMcdT0KpsgIkdFJNb3r2uMMaak+VK1NQenneRVd/p+d96YfNcARCQUmA0MxGmkXycicaqalKvoPFWdmGvd3kAfwPPQ4zdAf1VdCXTxKrcB+MSH72CMMSZAfEkk16jq1V7TX4nIRh/W6wnsVNXdACIyF6dKLHciyYsCEUAVQHAS2SHvAiLSBmgE/MeH7RljjAmQQqu2gEwRae2ZcJ9sz/RhvWbAPq/pFHdebsNFZJOILBDTdx3aAAAXm0lEQVSR5gCqugbnFuMD7muJqibnWm8kztVMxX+i0hhjyjBfEsljwAoRWSkiXwNfAb/zYb287hHOfdBfBES5/XYtx+nPCxG5EugAROIknxtEpF+udUcAH+X74SJjRWS9iKw/cuSID+EaY4wpDl/u2voSaANMcl/tgDM+bDsFaO41HQnsz7XtVFU9706+DnR338cA36nqaVU9DXwO9PKsJyJXA2GquqGAuF9T1R6q2qNhQ+tSwhhjAsWXKxJU9byqblLVje6Bf74Pq60D2ohIKxGpgnMFEeddQES8B2oYBniqr/YC/UUkzL1rrL/XMnCqtfK9GjHGGFN6fGlsz0uhj7araoaITASWAKHAW6qaKCJPA+tVNQ6YJCLDgAzgGDDKXX0BcAOwGac67AtVXeS1+V/hPGlvjDEmyIrV+6+I7FXVFgGIJyCs00ZjjCk6vzttFJFFXNo4Ds7ViA1+YIwxBii4auv/FnOZMcaYSiTfRKKqX5dmIMYYY8onn+7aMsYYY/JjicQYY4xfLJEYY4zxS6HPkeRz99YJYD3wd1W1Qa6MMaYS8+WKZDdwGqcLk9eBkzg98bZ1p40xxlRivjzZ3lVVvTtMXCQiq1S1n4gkBiowY4wx5YMvVyQNRST7KXb3fQN38kJAojLGGFNu+HJF8jvgGxHZhfNUeytgvIhUx+323RhjTOVVaCJR1cXuaITtcRLJVq8Gdhsv3RhjKjlf7toKB34DeNpJVorI31U1PaCRGWOMKRd8qdqagzNm+qvu9P3uvDGBCsoYY0z54UsiuUZVr/aa/kpENgYqIGOMMeWLL3dtZYpIa8+EiFwBZAYuJGOMMeWJL1ckjwErRGQ3TmN7S2B0QKMyxhhTbvhy19aX7l1b7XDv2gK6BDowY4wx5YNPY7ar6nlgk2daROYD5WaoXWOMMYFT3N5/pUSjMMYYU24VN5HkNZa7McaYSijfqq18uo8H52qkfsAiMsYYU64U1Ebyf4u5zBhjTCWSbyJR1a9LMxBjjDHlkw21a4wxxi8BTSQiMkhEtonIThF5PI/lo0TkiIgkuK8xXsteEJFEEUkWkZkiIu78KiLymohsF5GtIjI8kN/BGGNMwXx6jsRDRBqr6kEfy4YCs4GBQAqwTkTiVDUpV9F5qjox17q9gT5AtDvrG6A/sBL4E3BYVduKSAhQryjfwRhjTMkqUiIBFgPdfCzbE9ipqrsBRGQucDuQO5HkRYEIoArOXWLhOOPEAzyIMzYKqpoFHPU1eGOMMSWvqFVbRXkQsRmwz2s6xZ2X23AR2SQiC0SkOYCqrgFWAAfc1xJVTRaROu46fxWReBGZLyKXF/E7GGOMKUFFTSSvF6FsXkkn93Mpi4AoVY0GluMO3SsiVwIdgEic5HODiPTDuYKKBL5V1W7AGvK5FVlExorIehFZf+TIkSKEbYwxpiiKlEhU9dXCS2VLAZp7TUcC+3NtL9XtxwucJNXdfR8DfKeqp1X1NPA50AtIBc4AC91y88mnqk1VX1PVHqrao2HDhkUI2xhjTFEE8q6tdUAbEWklIlWAEUCcdwERaeI1OQxIdt/vBfqLSJg71G9/IFlVFecqZoBb7kZ8a3MxxhgTIEVtbPeZqmaIyERgCRAKvKWqiSLyNLBeVeOASSIyDMgAjgGj3NUXADcAm3Gqw75Q1UXusqnAeyISCxzBxkYxxpigEuckv2Lr0aOHrl+/PthhGGNMuSIiG1S1R2Hl7Ml2Y4wxfrFEYowxxi+WSIwxxvjFEokxxhi/WCIxxhjjF0skxhhj/GKJxBhjjF8skRhjjPGLJRJjjDF+sURijDHGL5ZIjDHG+MUSiTHGGL9YIjHGGOMXSyTGGGP8YonEGGOMXyyRGGOM8YslEmOMMX6xRGKMMcYvlkiMMcb4xRKJMcYYv1giMcYY4xdLJMYYY/xiicQYY4xfLJEYY4zxiyUSY4wxfgloIhGRQSKyTUR2isjjeSwfJSJHRCTBfY3xWvaCiCSKSLKIzBQRceevdLfpWadRIL+DMcaYgoUFasMiEgrMBgYCKcA6EYlT1aRcReep6sRc6/YG+gDR7qxvgP7ASnf6XlVdH6jYjTHG+C6QVyQ9gZ2qultVLwBzgdt9XFeBCKAKUBUIBw4FJEpjjDF+CWQiaQbs85pOceflNlxENonIAhFpDqCqa4AVwAH3tURVk73Wedut1vofT5WXMcaY4AhkIsnrAK+5phcBUaoaDSwH3gUQkSuBDkAkTvK5QUT6uevcq6qdgb7u6/48P1xkrIisF5H1R44c8fvLGGOMyVsgE0kK0NxrOhLY711AVVNV9bw7+TrQ3X0fA3ynqqdV9TTwOdDLXecn9+cp4EOcKrRLqOprqtpDVXs0bNiwhL6SMcaY3AKZSNYBbUSklYhUAUYAcd4FRKSJ1+QwwFN9tRfoLyJhIhKO09Ce7E43cNcNB4YCWwL4HYwxxhQiYHdtqWqGiEwElgChwFuqmigiTwPrVTUOmCQiw4AM4Bgwyl19AXADsBmnOuwLVV0kItWBJW4SCcWpDns9UN/BGGNM4UQ1d7NFxdOjRw9dv97uFjbGmKIQkQ2q2qOwcvZkuzHGGL9YIjHGGOMXSyTGGGP8YonEGGOMXyyRGGOM8UvAbv+tEOL/AapQtyXUaQm1IyE0PNhRGWNMmWKJpCDfzIBjuy9OSwjUiryYWHL/rHE5hNhFnjGmcrFEUpAJ6+DUfvj5Rzj+Y86fu76EUwdylg+tCnVa5J9oLqsL1sekMaaCsURSkNAwJzHUaYHTP2Qu6efgxD43wezJmWh+2gBnf85ZvmqtvBOM52eVaqXxrYwxpkRZIvFHeAQ0aOO88nLuBBzfCz/vyZlkUnfCzi8h42zO8tUbXkwsdaNyJhlrnzHGlFGWSAIpojY07uy8clOFtCNeCWZPzquZpH9BVsbF8tY+Y4wpoyyRBIsI1GjkvJpfc+nyzAxrnzHGlAuWSMqqUm2faQFVqpfGtzLGVECWSMorn9tncl3RpO6EXV9B+pmc5b3bZ3L/rN3c2meMMfmyRFJRlXj7TLP8bwSw9hljKjVLJJWRX+0zX1n7jDEmB0sk5lLWPmOMKQJLJKboit0+s8vaZ4ypgCyRmJJXaPvM0YttM0Vtn8n9s0Zja58xJsgskZjSJQI1GjqvyDyGgi5W+0zzfBJNlLXPGFMKLJGYssWf9pn98dY+Y0wQWCIx5Uuh7TMnL72SsfYZYwLKEompWCJq+d4+40k0P++x9hlj/GCJxFQehbXPZGXCyZ+sfcaYIrJEYoxHSGjJts9UqZn/Q5p1W1r7jKkwLJEY46vits8c2w27V1zaPlOtgdNGExLqXLlICOD+zPMlF8t5v3Ksk9/7XNvJ93MKWD/PdfKJKcc6+a1XlLh8/f65t5/P9vKNK691CvvdeH1WJRXQRCIig4CXgVDgDVWdnmv5KOBF4Cd31iuq+oa77AVgCBACLAMmq6p6rRsHXKGqnQL5HYzxWXHaZ86kusuzcr300umsTCD3fO9XXtvJyrVOfu9zzcv3c9RZZvJWaJIrJMH5lOR9SaZeZe/5GMKqBvRrByyRiEgoMBsYCKQA60QkTlWTchWdp6oTc63bG+gDRLuzvgH6Ayvd5XcApwMVuzElrrD2mfJENe8kdEny0QISlnf5AhKazwnOU6YocRUWWxHiKiz5FroPfI2LQr6/10mHprvrBP5KKZBXJD2Bnaq6G0BE5gK3A7kTSV4UiACq4OyFcOCQu50awKPAWODjkg/bGFMgTxUTdsdaWZeVpc6vK8CfE8hE0gzY5zWdAlybR7nhItIP2A48oqr7VHWNiKwADuDsg1dUNdkt/1fg/wFn8thWNhEZi5NsaNGihV9fxBhTsakq6ZlKRlYW6ZlKZpaSkZlFuvszI0vJyFTS3feZbrmMTCU9K4uMTK95ubfhzvNsw7PdzCx3e57lmUpGjnkX53s+N8MTj/u5mbniyhFrVhaqsO2ZQVQNCw3o/gtkIskrCeauXF0EfKSq50Xkt8C7wA0iciXQAYh0yy1zk81J4EpVfUREogr6cFV9DXgNoEePHlapa0wJUNUcB6pMrwNphtcBM8fBMPPiPOfgmfMAmZmlFw/YXut45nmvk+4esJ3P9z5w+vAZuQ7Y3p+RVcpHiPBQISwkhLAQISxUCAsNITxECA0VwkNCnHnZP53lEeEhhIY45TzrhIU45cJD5eI6bnnPZ0g5r9pKAZp7TUcC+70LqGqq1+TrwPPu+xjgO1U9DSAinwO9gFNAdxHZgxN7IxFZqaoDAvEFjCkOz8E2MyvngdQzneke0HIc3C45W714xnvx4OeZl/PsNN9teB/k8zjjdQ7gOQ+2OZNAXmfipXvEDQsRQkOE8NCLB9fwUK95IV4HVPcgHB4aQkT4xeXhoSGEei33Pjh75jnbu7gtzzrhOQ7obvlcB+z8tuEd78WDvhO7VLA7vAKZSNYBbUSkFc5dWSOAe7wLiEgTVfU85TUM8FRf7QUeEpHncK5s+gOxqroImOOuGwV8ZkmkbFJVspQcB6jMrIsHUM+BzHu+50CXo1z2gffS9T1VDDkO0tnb9j7Y5lMu6+LZrees2Hv6knJe2y8ovtI+uwUI9Rxwvc5GQ0PyOJB5H9xCQqgaHpbjDLawM96Ly70+IzSPeXkcQL2XF3yQv3iwr2gH3IoqYIlEVTNEZCKwBOf237dUNVFEngbWq2ocMElEhgEZwDFglLv6AuAGYDNOddgXbhIpd7LyOogWcHaab7miHvC861gvOdjlcWD1OgPNeaDPKxa9NEFckjCCX5vofVDzHJg806FeZ4eeA1loyMWDXtXwEKq5095lnZ/u/FDnwB3qbj/fcl4H8LAcnyfZB2rvKg3PQd77TDw8u8zFs2DPumEhQkiIHXBN8IjXoxkVVo8ePXT9+vVFXu+x+RvZe+zMJQfISw+aZevs1Jv3marn7PHSA57XwdXrAJX7gJfXQdhzxpk9LzSPg2he5XIdvL3nh18SY/6x5Jcg7MBqjP9EZIOqFnq/uj3ZXgDPWXV+Z6eXHvDyPzu99ECa99lpaCFnz2G5DtRhofmUq6B1scaYsscSSQFm3N0l2CEYY0yZZ08UGWOM8YslEmOMMX6xRGKMMcYvlkiMMcb4xRKJMcYYv1giMcYY4xdLJMYYY/xiicQYY4xfKkUXKSJyBPixmKs3AI6WYDglxeIqGouraCyuoqmIcR0FUNVBhRWsFInEHyKy3pe+ZkqbxVU0FlfRWFxFU9njsqotY4wxfrFEYowxxi+WSAr3WrADyIfFVTQWV9FYXEVTqeOyNhJjjDF+sSsSY4wxfqm0iUREmovIChFJFpFEEZmcRxkRkZkislNENolIN69lD4jIDvf1QCnHda8bzyYRWS0iV3st2yMim0UkQUSKPiykf3ENEJET7mcniMhfvJYNEpFt7r58vJTjeswrpi0ikiki9dxlgdpfESKyVkQ2unFNy6NMVRGZ5+6T/4pIlNeyJ9z520TkllKO61ERSXL/vr4UkZZeyzK99mVcKcc1SkSOeH3+GK9lgfp/9CWuGV4xbReR417LArK/vLYfKiLfi8hneSwrvb8vVa2UL6AJ0M19XxPYDlyVq8ytwOeAAL2A/7rz6wG73Z913fd1SzGu3p7PAwZ74nKn9wANgrS/BgCf5bFuKLALuAKoAmzMvW4g48pV/jbgq1LYXwLUcN+HA/8FeuUqMx74m/t+BDDPfX+Vu4+qAq3cfRdainFdD1Rz34/zxOVOny7pfVWEuEYBr+SxbiD/HwuNK1f5h4G3Ar2/vLb/KPBhPv93pfb3VWmvSFT1gKrGu+9PAclAs1zFbgf+oY7vgDoi0gS4BVimqsdU9WdgGVDoQzslFZeqrnY/F+A7ILIkPtvfuArQE9ipqrtV9QIwF2ffBiOukcBHJfHZhcSlqnranQx3X7kbJG8H3nXfLwBuFBFx589V1fOq+gOwE2cflkpcqrpCVc+4k6X19+XL/spPIP8fixpXqfx9AYhIJDAEeCOfIqX291VpE4k395KvK87ZhrdmwD6v6RR3Xn7zSysub7/GuWryUGCpiGwQkbElHZMPcV3nVgN8LiId3XllYn+JSDWcA8w/vWYHbH+51Q4JwGGcA12+f1+qmgGcAOoT4P3lQ1zecv99RYjIehH5TkR+WVIxFSGu4W6V2wIRae7OKxP7y60CbAV85TU7YPsLiAX+AGTls7zU/r4qfSIRkRo4B5Ypqnoy9+I8VtEC5pdWXJ4y1+P8o0/1mt1HVbvhVHlNEJF+pRhXPNBSVa8GZgGfelbLY1Olvr9wqrW+VdVjXvMCtr9UNVNVu+Cc0fcUkU65w85rtQLml1ZcTnAi9wE9gBe9ZrdQ50npe4BYEWldinEtAqJUNRpYzsWz7TKxv3CqjxaoaqbXvIDsLxEZChxW1Q0FFctjXkD+vip1IhGRcJyDzweq+kkeRVKA5l7TkcD+AuaXVlyISDTOJe3tqprqma+q+92fh4GFlFCViC9xqepJTzWAqi4GwkWkAWVgf7lGkKvaIZD7y+szjgMrubS6JXu/iEgYUBs4RoD3lw9xISI3AX8Chqnqea91PPtrt7tu19KKS1VTvWJ5Hejuvg/6/nIV9PdV0vurDzBMRPbgVBXfICLv5ypTen9f/jSwlOcXTlb+BxBbQJkh5GxsX+vOrwf8gNOwV9d9X68U42qBU6/ZO9f86kBNr/ergUGlGFdjLj6b1BPY664XhtMA2oqLje0dSysut5znn6h6Ke2vhkAd9/1lwH+AobnKTCBnY+jH7vuO5GwM3U3JNbb7EldXnAbYNrnm1wWquu8bADsouZsmfImridf7GOA7930g/x8Ljctd1g7nxg0pjf2V67MHkHdje6n9fYVRefUB7gc2u/WfAH/EOUijqn8DFuPcubUTOAOMdpcdE5G/Auvc9Z7WnNUlgY7rLzh1na86bWdkqHP5fDmw0J0XBnyoql+UYlx3AuNEJAM4C4xQ5y83Q0QmAktw7uB6S1UTSzEucA48S1U1zWvdQO6vJsC7IhKKc+X/sap+JiJPA+tVNQ54E3hPRHbiJLkRbsyJIvIxkARkABM0Z3VJoON6EagBzHf3zV5VHQZ0AP4uIlnuutNVNakU45okIsNw9skxnLu4Av3/6Etc4DSyz3X/3j0Cub/yFKy/L3uy3RhjjF8qdRuJMcYY/1kiMcYY4xdLJMYYY/xiicQYY4xfLJEYY4zxiyUSY4wxfrFEYkwZIU6X9g2Kue4oEWlaEtsypqgskRhTMYwCmhZWyJhAsERiTC4iEiUiW0XkDXEGwvpARG4SkW/FGTipp/ta7Q4qtFpE2rnrPioib7nvO7vrV8vnc+qLyFJ3G3/HqzM9EblPnAGVEkTk7+6T1YjIaRH5fyISL86gUw1F5E6czhU/cMtf5m7mYbfcZhFpH8h9Zio3SyTG5O1K4GUgGmiP03vrL4Df43TBshXop6pdcbqsedZdLxa4UkRigLeB3+jFsT1yexL4xt1GHG63LiLSAbgbp2fiLkAmcK+7TnUgXp0ei78GnlTVBcB64F5V7aKqZ92yR91yc9y4jQmIytzXljEF+UFVNwOISCLwpaqqiGwGonA6gXxXRNrgdMEdDqCqWSIyCtgE/F1Vvy3gM/oBd7jr/VtEPIOV3YjTs+06t6+ry3DGwgBn7Il57vv3gYJ6O/Ys2+D5HGMCwRKJMXk77/U+y2s6C+f/5q/AClWNEWdArZVe5dsAp/GtzSKvzu4EeFdVnyjm+h6emDOx/3UTQFa1ZUzx1AZ+ct+P8swUkdo4VWL9gPpu+0V+VuFWWYnIYJxuxwG+BO4UkUbusnru6Hvg/M96tnkP8I37/hTOmPXGlDpLJMYUzwvAcyLyLU7X+B4zgFdVdTvO6JXTPQkhD9OAfiISD9yMM34Lblfjf8YZAngTzhjkTdx10oCOIrIBuAF42p3/DvC3XI3txpQK60bemHJERE6rao1gx2GMN7siMcYY4xe7IjEmwERkNDA51+xvVXVCMOIxpqRZIjHGGOMXq9oyxhjjF0skxhhj/GKJxBhjjF8skRhjjPGLJRJjjDF++f/cLQ5yBWiRmQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_2.best_score_, gsearch2_2.best_params_))\n",
    "test_means = gsearch2_2.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_2.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_2.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_2.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_2.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_2.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(max_depth), len(min_child_weight))\n",
    "train_scores = np.array(train_means).reshape(len(max_depth), len(min_child_weight))\n",
    "\n",
    "for i, value in enumerate(max_depth):\n",
    "    pyplot.plot(min_child_weight, -test_scores[i], label= 'test_max_depth:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( '- Log Loss' )\n",
    "pyplot.savefig('max_depth_vs_min_child_weght_2_test.png' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -0.583831 using {'max_depth': 5, 'min_child_weight': 2}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAELCAYAAAAoUKpTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XmcFeWZ9//P1ftC000DERARRBRBFrVBcSHEqEF8RdS4onHLEx4XiGOiiZlJMmoyIyFOlEQyiriMUQMxxp/E6OPzqEEjRqUJIIs6AiFDBxBEaGig9+v3x1k4a5/TTZ9u6P6+X68jtdxVdZ1j9X1V3VV1l7k7IiIiLcnq7ABEROTQp2QhIiIpKVmIiEhKShYiIpKSkoWIiKSkZCEiIikpWYiISEpKFiIikpKShYiIpJTT2QG0lz59+vjgwYM7OwwRkcPKsmXLPnP3vqnKdZlkMXjwYCorKzs7DBGRw4qZ/T2dcmqGEhGRlJQsREQkJSULERFJqctcsxA5HDQ0NFBVVUVtbW1nhyLdTEFBAQMHDiQ3N7dNyytZiHSgqqoqSkpKGDx4MGbW2eFIN+Hu7Nixg6qqKoYMGdKmdagZSqQD1dbW0rt3byUK6VBmRu/evQ/qjFbJQqSDKVFIZzjY/U7NUCIincjdccA9cthpdsLD0fMODDcHh/NysigpaNu1iHRlNFmY2WRgDpANzHf3WUnKXQo8B4xz90ozywXmAycHY3zK3e/LZKwi0jW5O3CgYk2rYibwb3PEsEcNh8pBs8fPb064nchK/sDyofgORmlh7uGbLMwsG5gLnAtUAUvNbJG7r40pVwJ8C3gvYvJlQL67jzKzImCtmf3G3TdmKl6R7mDXrl08++yz3HLLLa1absqUKTz77LOUlZUlnN+aSraZ+Mp68tmTePm1xTSnWzHHbLM5YpnfL3iG1SuX8y8/+Vl4mZBbr72M+345n56lpZx2/EDe/bgq7rv88PZbmHjOVzj3gqlJfw/DMCP4MbII/GsGFjGcZZCVlYUZPPPEYxQWFXLZVVcH5hOYHxo2MxY88xQrl/+V2T+fE5gXsa77Z/+U737vrphlAsPXff3rLFtWSW5uLuPHj+eRRx5p811PyWTyzGI8sM7dNwCY2QJgKrA2ptyPgdnAHRHTHCg2sxygEKgHdmcwVpFDWrIj4paObmObKhz4e9Wn/OKhuXztmhujKt7GxkaysrKTHh3PeWIh2+rg0627kyaDgzH/+VfYXL0/alpWuPKNqJg5UAkfGM4iB4LDRkFOFnk5WfQqzotb/oUXXzpQwRsMKi8KD4cq5pLCHPqXFnLcESVR24leV+vb///5jm+lLFNSkEtBbjblxXlx8/5j9izu+dEPEi53zTVX88wzTwMwbdo05s+fz80339zqGFuSyWRxJLApYrwKODWygJmdBBzl7i+ZWWSy+B2BxLIFKAJud/fPMxirSFhTs1Pf2Ex9YzN1TU3h4fqmZuoaAv+G5zceGK9rjCgbNT0w3NDYzEVD4O879uIOv3zjE9Ztq4muZoOVd3gk9N806+IhfYv55lnHJJ3/L//8fTZu2MAXJ4wjJzeXoqJi+h7Rj4/WrOKPb77PjBumsXXLP6irreW6b97MVdfdSJbBxIoT+cNrb7Fv3z6uv/xixp02gWVL36Nf/wE8+exvKSoqiquYszC+ev65jB4zlpXL/8pnn21n3mNP8B8/m83a1au59LLLuOfeH2NmlJf1ZFf1bt56803uvfce+vTpw+rVqznllFN4+umnk1bOS5cu5bbbbmPv3r3k5+fz+uuvU94jn907tnHjlRezfv16Lr74YmbPng0c6EOud58+AJQV5eHuzJw5kzfeeIMhQ4aErwEU5GbHbe/9999n1qxZ/P73v+fFF1/kyiuvpLq6mubmZkaMGMGGDRtYv349t956K9u3b6eoqIhHH32U4cOHc/fdd9OjRw/uuOMOli5dyje+8Q2Ki4s588wzeeWVV1i9ejUAmzdvZvLkyVGx33XXXezfv5+xY8cycuRInnnmmai4pkyZEh4eP348VVXxZ0wHK5PJItH/3fAub2ZZwAPA9QnKjQeagAFAL+DPZvZa6CwlYh3TgekAgwYNap+opcM1N3ugIo6pZA9Uuk3UNTRTFzc9vpKOLZNonXVNzdQ1NEVvJ2K4sfng25AhUGHmZQeOcvNzssjLzmLKUeXUNTRjFjgih+g/FMs6MMVCcyxBudAGYqaXFuQytG+P8NFvbFPH3Af/g6kX/jerVn3Am2++yQUXXMDq1avD997/9tmnKC8vZ//+/YwbN46bb7ia3r17k5NlHFVeTE2e87cN63jutwsYO3Ysl19+OUtee5lrrrkm4W+Qk2WUFBWw5O0/M2fOHK667GssW7aM8vJyhg4dyp13fIfevXsHymZnkZVlLF++nDVr1jBgwADOOOMMlixZwplnnhm37vr6eq644goWLlzIuHHj2L17N4WFhQCsWLGC5cuXk5+fz/HHH8/MmTM56qijEsb4wgsv8PHHH7Nq1So+/fRTRowYwY033piw7Mknn8zy5csB+POf/8yJJ57I0qVLaWxs5NRTA8fC06dP5+GHH2bYsGG899573HLLLbzxxhtR67nhhhuYN28ep59+OnfddVfUvESxz5o1i4ceeogVK1aEy02ZMoX58+czYMCA8LSGhgZ+/etfM2fOnITxH4xMJosqIPL/zkBgc8R4CXAisDh41NAPWGRmFwLTgP/j7g3ANjNbAlQAUcnC3ecB8wAqKira5y+8G3D3+KPj2CPh8JF04kq1LrJcsEJPdAQeXSbx0XdDU/v9r8vLySI/WEGHPvmh4eD00rxc8nrkk58bUzY7cpnsA8MxZfJzY8tmkZedHb2tnCxysizuiPjDDz/kuH4lAPz8irHt9r3TlZ0VTDDBuMaPHx/1kNYvfvELXnjhBQA2bdrEJ598Eq7MQ4YMGcLYsYHYTznlFDZu3NjiNi+88EIARo0axciRI+nfvz8AxxxzDJs2bYpb//jx4xk4cCAAY8eOZePGjQmTxccff0z//v0ZN24cAD179gzP+/KXv0xpaSkAI0aM4O9//3vSZPHWW29x1VVXkZ2dzYABAzj77LOTfpecnByOPfZYPvzwQ95//32+/e1v89Zbb9HU1MRZZ51FTU0N77zzDpdddll4mbq6uqh17Nq1iz179nD66acDgWajl156qdWxv/zyy3HTbrnlFiZOnMhZZ52V9Du0VSaTxVJgmJkNAf4BXEkgCQDg7tVAn9C4mS0G7gjeDfVl4Gwze5pAM9RpwIMZjDWj3J2GJo87Yq5rbIpqxkhWqcaWia2AI8u0VLlHTmsveUkq28hKtaQgJ6rCjqyII5fLj6lsQxVw7PryEy6bTW52fOUsLSsuLg4PL168mNdee42//OUvFBUVMWnSpIQPceXn54eHs7Oz2b9/f1yZROWzsrKils3KyqKxsTHl+hOVgcDfVbL/3+muI6Q1+81ZZ53FK6+8Qm5uLueccw7XX389TU1N3H///TQ3N1NWVhZ1BpAo7pa0NvaQe+65h+3bt/PII4+k90VaKWPJwt0bzWwG8CqBW2cfd/c1ZnYvUOnui1pYfC7wBLCawFn2E+7+QSbi3LWvnr+s3xE+Co6qVGOOmEPzEjWHJDpiDq2voamZFPtH2nKyLO4INlBhZoePgnvk55BXlLhCjT4aTn70HXnEHHskHS4XbDaQw0dJSQl79uxJOK+6uppevXpRVFTERx99xLvvvtvB0bXO8OHD2bx5M0uXLmXcuHHs2bMn3AzVGhMnTuSRRx7h2muvZdu2bfzpT39i2rRpLZa/9tprufbaa+nbty87duxg69atjBw5EjNjyJAhPPfcc1x22WW4Ox988AFjxowJL9+rVy9KSkp49913Oe2001iwYEFacebm5tLQ0JDwLqf58+fz6quv8vrrr5OVlZlnrTP6nIW7vwy8HDPtR0nKTooYriFw+2zG/X3HPm5+5q9J52dnWdJKNTRckJtFaWFuwgo1/mj5QMUeub7YppPAkXR2XJu3Kmc5GL179+aMM87gxBNPpLCwkCOOOCI8b/LkyTz88MOMHj2a448/ntNOO60TI00tLy+PhQsXMnPmTPbv309hYSGvvfZaq9dz8cUX88YbbzBq1CiOO+44vvjFL7ZY/tRTT+XTTz9l4sSJAIwePZovfOEL4bOTZ555hptvvpmf/OQnNDQ0cOWVV0YlC4DHHnuMb37zmxQXFzNp0qRws1NLpk+fzujRozn55JN55plnoq5Z3HTTTRx99NFMmDABgEsuuYQf/ShhVdtm1h4PhBwKKioqvC1vyttf38TGHXtjKvcDlXm2KmdpRx9++CEnnHBCZ4chnaympoYePXoAMGvWLLZs2ZKRi9KxEu1/ZrbM3StSLdvtu/sozMvmhP49UxcUEWknf/zjH7nvvvtobGzk6KOP5sknn+zskFLq9slCRA7erbfeypIlS6Km3Xbbbdxwww3tsv6LL76Yv/3tb1HTfvrTn/KVr3ylXdbf0du84ooruOKKKw56PR1JyUJEDtrcuXMzuv7Q7bwdqTO2eShTF+UiIpKSkoWIiKSkZCEiIikpWYiISEpKFiLdyK5du/jVr37V6uWmTJnCrl27MhBRQKifpPbw5JNPMmPGjITzIr9H6DmHWNdffz2/+93v2i2ekIcffpinnnqqxTItxf7v//7vSZf7xje+wZgxYxg9ejSXXnopNTU1BxVrIkoWIt1IsmTR1NTU4nIvv/xy0hcftYd33nknY+uOlOnv0ZKbbrqJa6+9ts3Lt5QsHnjgAVauXMkHH3zAoEGDeOihh9q8nWR066xIZ3nlLti6qn3X2W8UnJ/w7cUA3HXXXaxfv56xY8eSm5tLjx496N+/PytWrGDt2rVcdNFFbNq0idraWm677TamT58OHHgPRE1NDeeffz5nnnkm77zzDkceeSQvvvhi0j6ZJk2axEknncSyZcvYvn07Tz31FPfddx+rVq3iiiuu4Cc/+QkQOMqvqalh8eLF3H333Qf1PgtI/E6IyO/Rp0+4D9OE77NI5lB9n0Wox113Z//+/RnpUFNnFiLdyKxZsxg6dCgrVqzgZz/7Ge+//z7/9m//xtq1gRdYPv744yxbtozKykp+8YtfsGPHjrh1fPLJJ9x6662sWbOGsrIynn/++Ra3mZeXx1tvvcVNN93E1KlTmTt3LqtXr+bJJ59MuP7ly5fz4IMPsnbtWjZs2BD3sF9I6H0Wc+bMYeXKlbz22mtR77NYuHAhq1atYuHChWzatCnhOiD6fRaPPvpoi2c5yd5n8d5770W9z+KXv/wly5Yt4/7770/4CtsbbriBhx9+mL/85S9kZ0e/ZClR7LNmzaKwsJAVK1aEE8WUKVPYvHlz1Dr79evHRx99xMyZM5N+h7bSmYVIZ2nhDKCj6H0WXed9Fk888QRNTU3MnDmThQsXttvT8yE6sxDpxpK9z2LlypWcdNJJab3PItX7Frr6+yzefvtt3n77bSZOnBj1PovQ58MPP4yLuyVtfZ9FqPwVV1yR8myvLZQsRLqRrvo+C4A9e/a0qmINmThxIgsWLKCpqYktW7bwpz/9KWX5Bx98kAkTJoTfZ/HRRx8xcuRIevbsGX6fBQQSw8qVK6OWj3yfBdDq91nEcnfWrVsXHv7DH/7A8OHD01pnayhZiHQjke+zuPPOO6PmTZ48mcbGRkaPHs0Pf/jDw+p9FmPGjOHcc89NeCaUysUXX8ywYcMYNWoUN998c5veZzF69Oio91k89thjjBkzhpEjR/Liiy/GreOxxx5j+vTpTJgwAXdv1fssrr76auDANQt357rrrmPUqFGMGjWKLVu2tPu7LEDvsxDpUHqfhYDeZyEiImnQ+yxEpFvS+yxaR++zEJFuSe+z6PoyeoHbzCab2cdmts7M7mqh3KVm5mZWETFttJn9xczWmNkqMyvIZKwiIpJcxs4szCwbmAucC1QBS81skbuvjSlXAnwLeC9iWg7wNPB1d19pZr2B+HvGRESkQ2TyzGI8sM7dN7h7PbAAmJqg3I+B2UDkPW/nAR+4+0oAd9/h7i33dCYiIhmTyWRxJBDZIUtVcFqYmZ0EHOXuLxHtOMDN7FUz+6uZfTeDcYp0G+qivGt2UR4yc+bMpN/rYGUyWSR6fj78UIeZZQEPAN9JUC4HOBO4OvjvxWb25bgNmE03s0ozq9y+fXv7RC3ShamL8q7ZRTlAZWVlRhN6JpNFFRDZ+9VAYHPEeAlwIrDYzDYCpwGLghe5q4A33f0zd98HvAycHLsBd5/n7hXuXtG3b98MfQ2RriOyi/Jx48bxpS99iWnTpjFq1CgALrroIk455RRGjhzJvHnzwssNHjyYzz77jI0bN3LCCSfwzW9+k5EjR3Leeeexf//+pNubNGkSt99+OxMnTuSEE05g6dKlXHLJJQwbNowf/OAH4XKho+HFixczadIkLr30UoYPH87VV1/dYl9KS5cu5fTTT2fMmDGMHz8+3JVJqJvvYcOG8d3vHmiYCH2PSO7OjBkzGDFiBBdccAHbtm1Lur3333+fSy65BCDcNXt9fT21tbUcc8wxAKxfv57JkydzyimncNZZZ/HRRx8BcPfdd3P//feH4x49ejQTJkzgzjvv5MQTTwxvI1HskV2Uh57gjtTU1MSdd94Z7oo9EzJ56+xSYJiZDQH+AVwJTAvNdPdqINypvJktBu5w90ozWw9818yKgHrgiwTOQkS6jJ++/1M++vyjdl3n8PLhfG/895LOnzVrFqtXr2bFihUsXryYCy64gNWrV4d7nn388ccpLy9n//79jBs3jq997WtxvcJ+8skn/OY3v+HRRx/l8ssv5/nnn+eaa65Jus1QF+Vz5sxh6tSpLFu2jPLycoYOHcrtt98et/7ly5ezZs0aBgwYwBlnnMGSJUsS9job6qJ84cKFjBs3jt27d0d1Ub58+XLy8/M5/vjjmTlzZtJeZyO7KP/0008ZMWIEN954Y8Kyyboob2xsjOqi/OGHH2bYsGG899573HLLLbzxxhtR67nhhhuYN28ep59+OnfdFX2jaKLYZ82axUMPPcSKFSvC5aZMmcL8+fMZMGAADz30EBdeeGG4R99MyFiycPdGM5sBvApkA4+7+xozuxeodPdFLSy708x+TiDhOPCyu/8xU7GKdFfqovzw76J88+bNPPfccyxevDhp3O0how/lufvLBJqQIqcl7OHK3SfFjD9N4PZZkS6ppTOAjpKsi/KioiImTZqUVhflLTVDRZbvql2UX3/99TQ1NXH//fdHdVGeTHt3Ub58+XLWrVvHscceC8C+ffs49thjwz3Rthf1OivSjaiL8niHexflF1xwAVu3bmXjxo1s3LiRoqKidk8UoGQh0q2oi/J4h3sX5R1FXZSLdCB1US6gLspFRCQN6qJcRLoldVHeOuqiXES6JXVR3vXpArdIB+sq1wnl8HKw+52ShUgHKigoYMeOHUoY0qHcnR07dlBQ0PbXAqkZSqQDDRw4kKqqKtTxpXS0goKC8JPxbaFkIdKBcnNzo7rXEDlcqBlKRERSUrIQEZGUlCxERCQlJQsREUlJyUJERFJSshARkZSULEREJCUlCxERSUnJQkREUlKyEBGRlDKaLMxsspl9bGbrzOyuFspdamZuZhUx0weZWY2Z3ZHJOEVEpGUZSxZmlg3MBc4HRgBXmdmIBOVKgG8B7yVYzQPAK5mKUURE0pPJM4vxwDp33+Du9cACYGqCcj8GZgNRb1o3s4uADcCaDMYoIiJpyGSyOBLYFDFeFZwWZmYnAUe5+0sx04uB7wH3ZDA+ERFJUyaThSWYFn7ji5llEWhm+k6CcvcAD7h7TYsbMJtuZpVmVqn3A4iIZE4m32dRBRwVMT4Q2BwxXgKcCCw2M4B+wCIzuxA4FbjUzGYDZUCzmdW6+0ORG3D3ecA8gIqKCr16TEQkQzKZLJYCw8xsCPAP4EpgWmimu1cDfULjZrYYuMPdK4GzIqbfDdTEJgoREek4GWuGcvdGYAbwKvAh8Ft3X2Nm9wbPHkRE5DBhXeXF8RUVFV5ZWdnZYYiIHFbMbJm7V6Qqpye4RUQkJSULERFJSclCRERSUrIQEZGUlCxERCQlJQsREUlJyUJERFJSshARkZSULEREJCUlCxERSSmTHQmKiEhbNNZD3R6o2x3z7x6orT4wHJo34CQ47eaMhqRkISLSXpoaoiv32t3RlXpUpR+bCCLmNdam3lZWDuT3hPwSKOqTuvxBSpkszGwoUOXudWY2CRgNPOXuuzIdnIhIh2hqjK6sYyvx2kTzEiSExv2pt2XZUBCs5PN7Bj49joDexwamxc7LLznwKSg9MJxTAJboHXOZkc6ZxfNAhZkdCzwGLAKeBaZkMjARkZSamxI008QcwadqwqnbAw37Um/LsoIVdUSFXdwXyo+JqNAj5hVEVvQRFX9uYYdW8u0lnWTR7O6NZnYx8KC7/9LMlmc6MBHpwpqboL4mQVNNTGUe1VSToAmnYW8aG7MDFXWoAi8qh16DoyvzZJV7aLncosOykm8v6SSLBjO7CrgO+GpwWm7mQhKRQ1Zz84FKPratPa32+eC8+po0NmbRTTD5PaGgDEqPClbssZV7gqaa/BLILYYs3fh5sNJJFjcANwH/5u5/C74m9enMhiUi7aq5OXAUnrCpJqY9PqqpJratfg+QxgvT8nrEH5mXHhnfVNNSG31eD1Xyh5CUycLd1wLfAjCzXkCJu8/KdGAiArhD/d40jthj2+kTHM2nU8nnFse3t5f0j2/GaamNPq8HZGVn/KeRjpXO3VCLgQuDZVcA283sTXf/doZjEzl8uQdufwwfwbfijprYed6cenu5RfFNMr37xjTJxDTbxLbR5/WAbN1NL4mls2eUuvtuM/tfwBPu/q9m9kGmAxPpVI31wQq+Otgss/tAxV9bHTG8O3BRNnZabTU0N6TeTk5hfJNM8ZAE7fAttNHn91QlLxmXzh6WY2b9gcuBf2nNys1sMjAHyAbmJ2u+MrNLgeeAce5eaWbnArOAPKAeuNPd32jNtqUbC90zH1nJJ6zwW0gC6TwUFWqXL+gZOIIv7gu9h0ZPy4/4N7biz+sBOXmZ/z1E2kE6yeJe4FVgibsvNbNjgE9SLWRm2cBc4FygClhqZouC10Aiy5UQuCbyXsTkz4CvuvtmMzsxuP0j0/lCcpiLvG8+0dF6XaJpu6OnpXM7ZW5RRFNMsEIvGxQxrTSmwu8ZnwTULi/dSDoXuJ8jcNQfGt8AfC2NdY8H1gXLY2YLgKnA2phyPwZmA3dEbCPyOY41QIGZ5bt7XRrblc4Svq0y0dF6dQvNOBHT6vek3k52fnxFXtI/OK0seeUemQSydfe3SGukc4F7IPBL4AwCt1O8Ddzm7lUpFj0S2BQxXgWcGrPuk4Cj3P0lM7uDxL4GLFeiyDD3wFOscUfr1cmP4GPLpXMxNis3uiLP7xl4AjbuCL40psKPSAI5+R3zm4hIWDrNUE8Q6N7jsuD4NcFp56ZYLtGjjuF798wsC3gAuD7pCsxGAj8FzksyfzowHWDQoEEpwunCYu+8SXbRNe6ovjp6mje1vB3LiqnII5tuSuOTQEFpfBI4TLs6EOnu0kkWfd39iYjxJ83sn9JYrgo4KmJ8ILA5YrwEOBFYbIHKox+wyMwuDF7kHgi8AFzr7usTbcDd5wHzACoqKtK4ifwQ1eKdN4mO9BMkgab6FBux+Iq85wDIH57kaL40flpesSp6kW4qnWTxmZldA/wmOH4VsCON5ZYCw4JPfP8DuBKYFprp7tVAuF/d4PMcdwQTRRnwR+D77r4knS/SaRLdeZPoCL6lJJDunTeRR+mhO2/imm/KEh/h62lYETkI6SSLG4GHCDQZOfAOgS5AWhTsfHAGgTuZsoHH3X2Nmd0LVLr7ohYWnwEcC/zQzH4YnHaeu29LI97WaayHmq1Jmmx2pWjGacOdNwWlgQo98s6bgtIDF14TXpTVnTci0rnMvfWtN2b2T+7+YAbiabOKigqvrKxs/YKblsJj5ySfn1PQwt01pckr9/BF2RLdeSMihywzW+buFanKtfWxz28Dh1SyaLPeQ+HCh5InAd15IyLS5mTRda5yFpXDyV/v7ChERA5pbb3iefjeeSQiIq2W9MzCzJJ1XG9AYcYiEhGRQ07SZOHuJR0ZiIiIHLp0472IiKSkZCEiIikpWYiISEpKFiIiklI6XZQnuiuqGqgEvhN6X4WIiHRd6TyU93MCvcU+S+C22SsJ9BD7MfA4MClTwYmIyKEhnWaoye7+iLvvcffdwW7Bp7j7QqBXhuMTEZFDQDrJotnMLjezrODn8oh5epJbRKQbSCdZXA18HdgW/HwduMbMCgl0JS4iIl1cymsWwQvYX00y++32DUdERA5FKc8szGygmb1gZtvM7FMzez74ylMREekm0mmGegJYBAwAjgT+EJwmIiLdRDrJoq+7P+HujcHPk0DfDMclIiKHkHSSxWdmdo2ZZQc/1wA7Mh2YiIgcOtJJFjcClwNbgS3ApcANmQxKREQOLSmThbv/j7tf6O593f0L7n4RcEk6KzezyWb2sZmtM7O7Wih3qZm5mVVETPt+cLmPzewraX0bERHJiLZ2JPjtVAXMLBuYC5wPjACuMrMRCcqVAN8C3ouYNoJAtyIjgcnAr4LrExGRTtDWZGFplBkPrHP3De5eDywApiYo92NgNlAbMW0qsMDd69z9b8C64PpERKQTtDVZpNPNx5HApojxquC0MDM7CTjK3V9q7bIiItJxkj7BnaRrcgicVRSmse5EZx/h9ZlZFvAAcH1rl41Yx3RgOsCgQYPSCElERNoiabJw95KDXHcVcFTE+EACXZ2HlAAnAovNDALdni8yswvTWDYU4zxgHkBFRYU6NRQRyZBMvilvKTDMzIaYWR6BC9aLQjPdvdrd+7j7YHcfDLwLXOjulcFyV5pZvpkNAYYB72cwVhERaUE6Lz9qE3dvNLMZwKtANvC4u68xs3uBSndf1MKya8zst8BaoBG41d2bMhWriIi0zNy7RutNRUWFV1ZWdnYYIiKHFTNb5u4VqcplshlKRES6CCULERFJSclCRERSUrIQEZHWGyBNAAARv0lEQVSUlCxERCQlJQsREUlJyUJERFJSshARkZSULEREJCUlCxERSUnJQkREUlKyEBGRlJQsREQkJSULERFJSclCRERSUrIQEZGUlCxERCQlJQsREUlJyUJERFJSshARkZQymizMbLKZfWxm68zsrgTzbzKzVWa2wszeNrMRwem5ZvZfwXkfmtn3MxmniIi0LGPJwsyygbnA+cAI4KpQMojwrLuPcvexwGzg58HplwH57j4KOAX432Y2OFOxiohIyzJ5ZjEeWOfuG9y9HlgATI0s4O67I0aLAQ/NAorNLAcoBOqByLIiItKBcjK47iOBTRHjVcCpsYXM7Fbg20AecHZw8u8IJJYtQBFwu7t/nsFYRUSkBZk8s7AE0zxugvtcdx8KfA/4QXDyeKAJGAAMAb5jZsfEbcBsuplVmlnl9u3b2y9yERGJkslkUQUcFTE+ENjcQvkFwEXB4WnA/3H3BnffBiwBKmIXcPd57l7h7hV9+/Ztp7BFRCRWJpPFUmCYmQ0xszzgSmBRZAEzGxYxegHwSXD4f4CzLaAYOA34KIOxiohICzJ2zcLdG81sBvAqkA087u5rzOxeoNLdFwEzzOwcoAHYCVwXXHwu8ASwmkBz1hPu/kGmYhURkZaZe9xlhMNSRUWFV1ZWdnYYIiKHFTNb5u5xzfyx9AS3iIikpGQhIiIpKVmIiEhKShYiIpKSkoWIiKSkZCEiIikpWYiISEpKFiIikpKShYiIpKRkISIiKSlZiIhISkoWIiKSkpKFiIiklMnXqoqISAa4O/sb97Ozbie76nZRmF3IMWVxLxNtV0oWIiKdyN3Z17iPXXW72FW7K5wAdtXuCvxbt4udtTuprqsOzAtOr2+uD69j8uDJ/OyLP8tonEoWIiLtJFTx76zdGVXRh4ajEkDdTqprAwmgobkh4foMozS/lLL8MnoV9GJAjwGM7D2SsoKywLT8XpTllzGwZGDGv5uShYhIAu7O3oa9gUq9rjp5Aog58m9sbky4vizLojSvNFzRD+wxkFF9RlGaXxqu9ENJITRckldCdlZ2B3/zxJQsRKTLc3dqGmrCR/aJEkCiJNBSxR+q0MvyyxhUMoiyvgeO9kvzS6Mq/V4FvSjJKyHLDt97ipQsROSw4u7sadgTbsJpqV0/cnqjJ674sy073NQTqvjH9B1zIBkUxCeAw73ibwslCxHpNKGKP3RhN1FzT2wCSKfi75Xfi7KCMo7ueXS44o880o9MAN2x4m+LjCYLM5sMzAGygfnuPitm/k3ArUATUANMd/e1wXmjgUeAnkAzMM7dazMZr4i0XbM3s6d+T9TF3FTt/dV11TR5U8L1ZVt2uJIvzS9lSOmQcHt/ovb9soIySnJLMLMO/ubdQ8aShZllA3OBc4EqYKmZLQolg6Bn3f3hYPkLgZ8Dk80sB3ga+Lq7rzSz3kDi2wVEpN2FKv7YNvxkCSD0afbmhOvLsZyoin5o2dDwGUBc+37wrKBHbg9V/IeQTJ5ZjAfWufsGADNbAEwFwsnC3XdHlC8GPDh8HvCBu68MltuRwThFurRmb2Z33e6kF3ETNfdU11cnr/izcqKO7IeWDY072o9sCirLV8XfFWQyWRwJbIoYrwJOjS1kZrcC3wbygLODk48D3MxeBfoCC9x9dgZjFTksNDU3BY74Yx7caulBrlQVf2SlfmzZsUnv5gklgOLcYlX83VAmk0WivcnjJrjPBeaa2TTgB8B1wbjOBMYB+4DXzWyZu78etQGz6cB0gEGDBrVv9CIZ1tTcxO763YmbdZI8yVtdV43H/xkBkJuVG6joCwKV+nG9jkt4N0/kcFFOkSp+SUsmk0UVcFTE+EBgcwvlFwD/GbHsm+7+GYCZvQycDEQlC3efB8wDqKioSPwXJNKB3J3d9bvZuncrW/ZuCX921u6Mbu+v28nuut0pK/5QRX9cr+MS3s0TOVyYU6iKXzImk8liKTDMzIYA/wCuBKZFFjCzYe7+SXD0AiA0/CrwXTMrAuqBLwIPZDBWkbQ0NDXw6b5P2bJ3S1xC2FoTGN/XuC9qmZysHMoLysOVe7/yfknv5gmVUcUvh5qMJQt3bzSzGQQq/mzgcXdfY2b3ApXuvgiYYWbnELjTaSeBJijcfaeZ/ZxAwnHgZXf/Y6ZiFYHAWcGuul0HKv+9W9lSc2B4696tbN+/Pe5soLygnH7F/RhcOpgJAybQr7gf/Yv7Bz49+lNeUK77+OWwZ+5do/WmoqLCKysrOzsMOYTVNdXx6d5Po88GYhJCbVP0ozx5WXn079E/OgEUHxg/ovgICnMKO+kbiRy84PXgilTl9AS3dAnuzue1n8c3DUUkgx218Xdg9y7oTf/i/gzrNYyzBp4VlxDKC8rVHCSCkoUcJvY37g8ngmRnB5H9+wMUZBfQv0eg4j++/Pi4s4Mjio8gLzuvk76RyOFFyUI6XbM3s2P/jujKf++WqOahnXU7o5YxjL6FfenXox/Dy4fzpaO+FJcMSvNLdVYg0k6ULCTj9jXsS9w8FEwIW/dtjesKuiinKNAU1KMfI/uMjL9WUHQEudm5nfSNRLofJQs5KE3NTXy2/7PEt5IGx6vrqqOWybIsvlD0BfoX92dU31GcV3xeVCLoV9yPnnk9dVYgcghRspAW1dTXxJ8NhM4I9m5l275tcd1Fl+SW0K9HoOIf03dMXPNQ36K+5GRp1xM5nOgvthtrbG5k+77tiZuHgg+Z7WnYE7VMtmVzRNER9Cvux0lHnBTXPNSvuB8leSWd9I1EJFOULLqo0EtlQmcAiRLCtn3b4jqYK80vpX9xf47scSQVR1REnRX0K+5H38K+h8w7gUWk4yhZHKZiu52IOiMIDu9t2Bu1TE5WDv2K+tG/R3/G9xuf8EGzotyiTvpGInIoU7I4BLk71XXVLTYPJep2old+L/oV92NQySBO7X9qVPNQ/+L+9C7srW4nRKRNlCw6QX1TfeIHyyKG9zfuj1omLysvXPFPGDAh/LBZ5LUCdTshIpmiZNHOYrudSNQ89Nn+z+KWC3U7cWzZsZx55JnqdkJEDilKFq1U21ibMBFEXjuoa6qLWqYguyB8BjBx4MSE3U7kZ+d30jcSEUlNySJCszfzee3n4W4mEp0ZfF77edxyfQv7Bvof6nU8kwZOiuultCy/TGcFInJY6/bJ4pOdnzDr/VnhZNDQ3BA1vzCnkAHFA+jXox8jeo+Iu05wRJE6oxORrq/bJ4v87Hzqm+o5sfeJnHP0OXHXCtTthIiIkgWDeg7i11N+3dlhiIgc0nTTvYiIpKRkISIiKSlZiIhIShlNFmY22cw+NrN1ZnZXgvk3mdkqM1thZm+b2YiY+YPMrMbM7shknCIi0rKMJQszywbmAucDI4CrYpMB8Ky7j3L3scBs4Ocx8x8AXslUjCIikp5MnlmMB9a5+wZ3rwcWAFMjC7j77ojRYjjQM56ZXQRsANZkMEYREUlDJpPFkcCmiPGq4LQoZnarma0ncGbxreC0YuB7wD0ZjE9ERNKUyWSR6Ek2j5vgPtfdhxJIDj8ITr4HeMDda1rcgNl0M6s0s8rt27cfdMAiIpJYJh/KqwKOihgfCGxuofwC4D+Dw6cCl5rZbKAMaDazWnd/KHIBd58HzAMws+1m9veDiLcPEN8dbOdTXK2juFpHcbVOV4zr6HQKZTJZLAWGmdkQ4B/AlcC0yAJmNszdPwmOXgB8AuDuZ0WUuRuoiU0Usdy978EEa2aV7l5xMOvIBMXVOoqrdRRX63TnuDKWLNy90cxmAK8C2cDj7r7GzO4FKt19ETDDzM4BGoCdwHWZikdERNouo31DufvLwMsx034UMXxbGuu4u/0jExGR1tAT3AfM6+wAklBcraO4WkdxtU63jcvc425QEhERiaIzCxERSanLJwsz2xjR/1RlgvlmZr8I9l/1gZmdHDHvOjP7JPhp14vvacR1dTCeD8zsHTMbk+6yGY5rkplVB+evMLMfRcxrsS+wDMd1Z0RMq82syczK01n2IOMqM7PfmdlHZvahmU2Imd9Z+1equDpr/0oVV2ftX6ni6qz96/iI7a4ws91m9k8xZTpmH3P3Lv0BNgJ9Wpg/hUD/UwacBrwXnF5OoLuRcqBXcLhXB8Z1emh7BPrXei/dZTMc1yTgpQTTs4H1wDFAHrASGNFRccWU/SrwRgf9Xv8F/K/gcB5QdojsX6ni6qz9K1VcnbV/tRhXZ+1fCX6DrcDRnbGPdfkzizRMBZ7ygHeBMjPrD3wF+H/u/rm77wT+HzC5o4Jy93eC2wV4l8BDjYeylH2BdaCrgN9keiNm1hOYCDwG4O717r4rpliH71/pxNUZ+1eav1cyGdu/2hBXh+xfCXwZWO/usQ8fd8g+1h2ShQP/18yWmdn0BPOT9WGVVt9WGYwr0jeI7n23NctmIq4JZrbSzF4xs5HBaYfE72VmRQT+IJ5v7bJtcAywHXjCzJab2XwL9GsWqTP2r3TiitRR+1e6cXX0/pX279XB+1esK0mcpDpkH+sOyeIMdz+ZwKn2rWY2MWZ+sj6s0urbKoNxBYIz+xKBP+bvtXbZDMX1VwKnwWOAXwL/XyjUBOvq8N+LQBPBEnf/vA3LtlYOcDLwn+5+ErAXiG1L74z9K524AsF17P6VTlydsX+l/XvRsftXmJnlARcCzyWanWBau+9jXT5ZuPvm4L/bgBcInM5GStaHVWv7tmrvuDCz0cB8YKq772jNspmKy913e7CDRw88dJlrZn04BH6voLijrwz+XlVAlbu/Fxz/HYFKJ7ZMR+9f6cTVGftXyrg6af9K6/cK6sj9K9L5wF/d/dME8zpkH+vSycLMis2sJDQMnAesjim2CLg2eEfBaUC1u28h0E3JeWbWy8x6BZd9taPiMrNBwO+Br7v7f7fyO2Uyrn5mZsHh8QT2oR1E9AUWPAq6ksBv2yFxBeeVAl8EXmztsm3h7luBTWZ2fHDSl4G1McU6fP9KJ67O2L/SjKvD9680/z92+P4Vo6XrJB2zjx3M1flD/UOgLXJl8LMG+Jfg9JuAm4LDRuCNfuuBVUBFxPI3AuuCnxs6OK75BPrLWhH8VLa0bAfGNSM4byWBC6OnRyw/Bfjv4G/ZoXEFx68HFqSzbDvGNhaoBD4g0GTSq7P3rzTj6vD9K824Onz/Sieuztq/gtsoIpAwSyOmdfg+pie4RUQkpS7dDCUiIu1DyUJERFJSshARkZSULEREJCUlCxERSUnJQkREUlKyEOlgFujSuk8bl73ezAa0x7pEWkPJQuTwcj0wIFUhkfamZCHdlpkNtsDLbuZb4IU2z5jZOWa2xAIvixkf/LwT7I30nVCXEGb2bTN7PDg8Krh8UZLt9Daz/xtcxyNEdPBmZteY2fsWeLHNI2aWHZxeY2b/YWZ/NbPXzayvmV0KVADPBMsXBlczM1hulZkNz+RvJt2XkoV0d8cCc4DRwHBgGnAmcAfwz8BHwEQP9Eb6I+Dfg8s9CBxrZhcDTwD/2933JdnGvwJvB9exCBgEYGYnAFcQ6LV0LNAEXB1cpphAx3EnA28C/+ruvyPQJcXV7j7W3fcHy34WLPefwbhF2l1OZwcg0sn+5u6rAMxsDfC6u7uZrQIGA6XAf5nZMALdO+cCuHuzmV1PoC+hR9x9SQvbmAhcElzuj2YWeunQl4FTgKXBvvMKgW3Bec3AwuDw0wQ6/UsmNG9ZaDsi7U3JQrq7uojh5ojxZgJ/Hz8G/uTuF5vZYGBxRPlhQA3pXUNI1AmbAf/l7t9v4/IhoZib0N+0ZIiaoURaVgr8Izh8fWhisLvqOQTOGnoHryck8xbB5iUzO59Aj6YArwOXmtkXgvPKzezo4LwsILTOacDbweE9QMlBfB+RNlGyEGnZbOA+M1sCZEdMfwD4lQfeBfENYFao0k/gHmCimf2VwDsF/gfA3dcCPyDwSs4PCLwjuX9wmb3ASDNbBpwN3Buc/iTwcMwFbpGMUxflIocgM6tx9x6dHYdIiM4sREQkJZ1ZiLQTM7sBuC1m8hJ3v7Uz4hFpT0oWIiKSkpqhREQkJSULERFJSclCRERSUrIQEZGUlCxERCSl/x/ndrf8M7z0KQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_2.best_score_, gsearch2_2.best_params_))\n",
    "test_means = gsearch2_2.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_2.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_2.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_2.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_2.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_2.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(max_depth), len(min_child_weight))\n",
    "train_scores = np.array(train_means).reshape(len(max_depth), len(min_child_weight))\n",
    "\n",
    "#for i, value in enumerate(max_depth):\n",
    "#    pyplot.plot(min_child_weight, -test_scores[i], label= 'test_max_depth:'   + str(value))\n",
    "for i, value in enumerate(min_child_weight):\n",
    "    pyplot.plot(max_depth, -train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig('max_depth_vs_min_child_weght_2_train.png' )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "上面两个图可以得出，最佳的参数max_depth = 5,min_child_weight = 2  \n",
    "上一轮参数调优时，取值如下max_depth': [2,4,6,8,10], 'min_child_weight': [1,3,5,7]，得出的最优值分别为6和3，说明max_depth = 6的性能好于4，而本轮取值max_depth = [5,6,7],虽然5是边界值，但由于性能 4 < 6 < 5，故不需要在扩大边界重新调优，类似的，min_child_weight = 2也是最佳值，这两个值将作为下一轮参数调优的固定值  \n",
    "\n",
    "得到的最优logloss = 0.583831，比上一轮的调优结果0.584098 好一丢丢"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
